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Executive Summary 



This report presents the findings of the formative evaluation of the Understanding the Early 
Years (UEY) Initiative. The evaluation was conducted to examine issues of implementation 
and design, early progress in achieving immediate objectives, and issues related to 
accountability. The evaluation team was also asked to provide preliminary guidance on the 
design of a summative evaluation of the Initiative. 



Program Description 

The UEY Initiative is delivered by Human Resources and Skills Development Canada 
(HRSDC) to help communities generate and use local information about their children to: 

• identify gaps in services and programs for young children and their families, and 

• foster partnerships among community groups and individuals to make informed decisions 
about the best programs and services for young children to thrive. 

The UEY Initiative was announced in 2004 as a seven-year national initiative (2003-04 to 
2010-11) to support up to 100 communities across Canada. It provides three years of 
contribution funding to sponsoring organizations, which act in concert with their 
communities to use UEY research tools to collect local information on: 

• the development of kindergarten children; 

• family and community factors that influence child development; 

• programs and services for young children and their families; and 

• local socio-economic characteristics. 

This information allows members of funded communities to work together to identify 
gaps in services and programs for young children and their families and to develop a 
Community Action Plan to guide community decisions on how best to support families 
with young children. 



Evaluation Methodologies and Limitations 

The formative evaluation focused on the 21 communities selected for UEY funding through 
the 2005 Call for Proposals and was conducted approximately half way into the three-year 
funding agreements with these communities. The approach developed for the evaluation was 
based on the use of multiple lines of evidence drawn from three main sources: a review of 
documents and files (including a review of the administrative files for each of the 
21 UEY sites); a literature review; and key informant interviews with 44 individuals 
(program management and staff (n=6), provincial/territorial government representatives 
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and experts in childhood development (n=5), UEY community representatives (n=24), 
non-funded applicants (n=5), and others (n=4)). Findings from the Early Implementation 
Review were also used to augment the formative evaluation. 1 

Combined, these sources provide a useful information base from which to draw findings 
on the overall development of the UEY Initiative. Certain limitations should be noted, 
however. For example, staff turnover within UEY communities meant that it was 
sometimes difficult to reach the most suitable key informants. Another potential 
limitation is that key informant interviews were conducted with representation from only 
ten of the 21 UEY sites. The effects of this limitation were minimized, however, because 
the majority of data collection focused on all 21 sites (e.g. the file and document review 
and the other key informant interviews). 



Evaluation Findings on Issues of Design and 
Implementation 

i) UEY Objectives 

The document and file review indicated that the mandate, objectives and intended outcomes 
of the UEY Initiative are consistently communicated. This finding is consistent with the 
Early Implementation Review, which concluded that the various elements of strategic 
planning for the Initiative - including mandate, objectives, and intended outcomes - support 
the vision, goals and priorities of the department. 

The focus and measurability of some of the Initiative’s intended outcomes could be 
improved, particularly in the case of the longer-term and ultimate outcomes. 
Program officials who were key informants had mixed opinions on whether the Initiative’ s 
outcomes are well-defined and achievable. For example, a few felt that the ultimate goal 
(i.e. strengthened Canadian families with children by providing choice and support that 
recognizes the diversity of their needs) is very broad and should be more focused. 

Key informants indicated that participants appear to have a clear understanding of 
their project’s immediate objectives and expected results for the UEY Initiative. The file 
review and key informant interviews with program officials indicated that all 21 UEY 
communities have made progress in implementing their project and were making 
progress towards their immediate objectives and expected results. Although some early 
difficulties were experienced by most sites, the early difficulties were largely a result of 
the challenges/characteristics within individual communities (e.g. buy-in challenges 
and staffing) rather than a result of specific flaws in UEY design. 



1 HRSDC’s Internal Audit Services completed an Early Implementation Review of the UEY Initiative in 2006. 
The review examined whether management practices and systems were developed and aligned to support program 
objectives, and were operating as intended. 
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ii) Flexibility and Project Selection 

Evidence from the document and file review and key informant interviews indicated that 
the UEY model is inclusive of a broad range of communities and is sufficiently flexible 
to accommodate their participation. For example, the 21 UEY sites selected through the 
2005 Call for Proposals are very diverse in terms of population density, socio-economic 
status, ethno-cultural composition and languages. 

Looking at the process used to select projects for UEY funding, the Early Implementation 
Review concluded that the assessments conducted by provincial/territorial government 
representatives and early childhood development experts are intended to ensure objectivity 
in decision-making and minimize the risk of error, abuse or fraud. The formative 
evaluation found that the UEY proposal review process is generally considered to work 
well and to adequately include participation of regional representatives and experts. 
Key informants familiar with the process noted a good balance between UEY staff and 
provincial/territorial representatives. The experience of non- funded applicants suggested, 
however, that the mandatory requirements ( which are intended to ensure the capacity of 
the community to be a UEY site and the integrity of the data ) were not consistently well 
understood by all applicants. 



Hi) Roles and Responsibilities 

The evidence from key informants indicated that the roles and responsibilities of key 
players in the UEY Initiative are well understood and have been implemented as 
intended , although clarification was provided to respond to some initial challenges. 



iv) UEY Design and Key Research Methods 

The key strengths of the UEY design are considered to include collaboration, community 
focus, credibility and communication. The main design concern identified by key 
informants was a lack of sufficient linkage between two of the main sets of research data. 
Improvements suggested for UEY design focused on improving or refining processes, 
rather than on a redesign of the initiative (e.g. improving coordination with 
provincial/territorial programs and projects). Many of the suggestions have been a 
focus of the UEY Initiative from the beginning. 

The UEY Initiative uses three data collection methods: the Early Development Instrument 
(EDI), the Parent Interviews and Direct Assessments of Children Survey (PIDACS), and 
data collection and analysis for the Community Mapping Report. 

Key informants indicated that a key strength of EDI is that it is a standardized instrument 
that is nationally used and recognized. Most of the challenges/limitations identified for 
the EDI were related to the EDI process (e.g. certain aspects of communication, reporting 
and data sharing), rather than related to the EDI instrument. The suggested improvements 
were largely related to these challenges. 
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Similarly, a key strength of the PIDACS is that it is a validated, standardized instrument 
based on the National Longitudinal Survey of Children and Youth. Another key strength 
is that the instrument can be tailored and customized to accommodate the differences 
across UEY sites. The most frequently cited challenge was the delayed implementation 
of the PIDACS. A particular concern was that, because of the delay, the cohort for the 
PIDACS data is different from the cohort for the EDI data. Some of the suggested 
improvements for the PIDACS tended to be similar to the suggestions made for the EDI. 
Other suggestions were process changes specifically related to the PIDACS (e.g. adding 
further flexibility to accommodate differences across communities, and hiring 
local/known people as PIDACS administrators). 

Key strengths of the Community Mapping Report are the provision of useful data and 
information, and the development of local capacity to undertake research. In this case, 
challenges/limitations related to capacity issues within the community (i.e. a limited capacity 
to develop the report) and challenges in obtaining relevant data. A few key informants 
suggested the community mapping exercise could be improved by encouraging information 
sharing among UEY communities and by providing more/better guidance and assistance 
when communities are preparing the Mapping Report. 

The document review indicated that the UEY Initiative has been pro-active in 
responding to challenges/limitations in design and delivery. This included working to 
resolve process-related issues in the areas of planning/delays, clarity/guidance/tools and 
communication. 



Progress Toward Immediate Objectives 

i) Increased Support for Communities 

Evidence from key informants and the file review suggests that progress has been made 
in providing communities with funding support. Although it is too early to assess the 
ability to attract additional support, the file review indicated that the UEY Initiative has 
enabled community coalitions to identify priorities for grants, incorporate UEY into 
funding proposals, and apply for and secure funding. The evidence also indicated that 
progress has been made in providing communities with technical assistance, with 
community mapping being cited as a main catalyst. 



ii) Increased Availability and Sharing of Information 

Both the project file review and key informants revealed that UEY communities have 
been successful at forming or strengthening partnerships as a result of the Initiative. 
As well, evidence from the document and file review and key informants suggests that 
the Initiative has increased the number of opportunities and tools for knowledge 
sharing and networking within communities on the role of families and communities 
in child development. 
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iii) Increased Awareness Regarding Child Development 

Most key informants who were program officials felt there has been progress in terms 
of increasing parental and community awareness and knowledge regarding child 
development , although it was too early for this progress to be fully evident. Cited examples 
of progress included the creation of brochures and presentations at workshops. 



Issues Related to Accountability 

Despite the array of provincial programs aimed at children, the program scan conducted 
for the formative evaluation found no evidence of overlap or duplication with the 
UEY Initiative In addition, a number of safeguards have been put in place to avoid or 
reduce the potential for overlap and duplication (e.g. UEY officials provide program 
updates at the semi-annual meetings of the Federal/Provincial/ Territorial Committee on 
Early Childhood Development Knowledge Information and Effective Practices). 

The general view among key informants is that the results-based reporting strategies for 
the UEY Initiative are appropriate and compatible with federal government accountability 
requirements. Most key informants who were community representatives expressed 
confidence in their community’s capacity to undertake the necessary reporting, although 
most also expressed the desire for more clarity and feedback from HRSDC to support 
reporting on the results of data collection. All key informants who were program officials 
felt that the performance monitoring reports of UEY sites were being sent in a timely manner. 
The file review found, however, that two-thirds of project files were missing at least one 
quarterly report. 



Some Preliminary Guidance for the Summative Evaluation 

A number of features of the UEY Initiative (e.g. a multi-site funding framework with 
community-level interventions) were identified as important considerations when determining 
how best to conduct a summative evaluation. Taking into account these features and the 
uniqueness of each UEY community, the evaluation team suggested that a theory of change 
approach ' would be the most appropriate for the summative evaluation. 

There is evidence that not all of the baseline data necessary for the summative evaluation 
is being collected consistently. For example, there is a need to define more clearly the 
information/data being requested from UEY sites (e.g. to clarify what is meant by 
partnerships). As well, measures identified in the Performance Measurement Framework 
for the UEY Initiative should ideally be designed to be on-going, rather than providing a 
single snapshot. 



2 A Theory of Change model creates a commonly understood vision of a program or initiative’s long-term goals, how they 
will be reached (the types of interventions applied) and what will be used to measure progress along the way. A Theory 
of Change is a specific and measurable description of a social change initiative that forms the basis for strategic planning, 
on-going decision-making and evaluation. Each outcome identified in the pathway of change is tied to an intervention and 
its necessary preconditions, revealing the often complex web of activity that is required to bring about change. 
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Overall Conclusions 

The formative evaluation found the design of the UEY Initiative to be generally sound. 
For example, the evidence showed that the UEY model is inclusive, as well as sufficiently 
flexible to allow for the participation of a broad range of communities. Collaboration, 
community focus, credibility and communication were cited as key strengths of the 
UEY design. 

The evaluation found that the EDI and PIDAC are viewed positively as standardized tools 
that are recognized nationally. Key strengths of the Community Mapping Report include 
the ability to provide useful data and to develop local capacity to undertake research. 

The evaluation found that participants appear to have a clear understanding of their 
project’s immediate and expected results for the UEY Initiative and have made progress in 
implementing their project. The early evidence suggests that progress has been made 
towards providing the community with funding support and technical assistance. As well, 
there is evidence that the UEY communities are forming and strengthening partnerships. 
There is also evidence of progress in terms of increasing parental and community 
awareness and knowledge regarding child development. 

The following areas were identified for consideration/improvement: 

• Some of UEY's long-term objectives may be too broad relative to the scope and size of 
the UEY Initiative. These objectives should be clarified or modified to better link them 
to the resources and scope of the UEY Initiative. 

• The UEY Initiative is working to resolve process-related issues (e.g. related to planning, 
delays) and this work should continue. 

• Consider streamlining parent interviews. 

• The linkage of research data from EDI and PIDACS may better enable research on 
possible relationships between early childhood outcomes and a number of family and 
community factors. As well, there is a need for program officials to better communicate 
with community coordinators and others in the community about the cohort used for 
the PIDACS and the usefulness of these data. The continued concern expressed by 
community representatives in this area indicates the need to validate the PIDACS results 
once they are all available. 

• It would be helpful to further assist communities in completing the Community Mapping 
report. This could be accomplished in different ways (e.g. additional in-person training 
sessions, an on-line training module or a helpline). 

• To better support the summative evaluation, the Initiative needs to consistently collect 
data at the community level on how knowledge gained within the UEY communities is 
being applied regarding child development. Guidance will need to be provided to the 
communities on how best to collect these data. 



VI 
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• To better support the summative evaluation, the UEY Initiative and the UEY sites should 
ensure that outcomes are linked to measurable indicators and that the necessary data can, 
and are, consistently collected. Consideration should be given to improving the activity 
report template to better facilitate the quantification and analysis of UEY project 
progress/performance. As well, there is a need for improved quantitative baseline data 
for indicators of intermediate and longer-term outcomes. 
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Management Response 



Introduction 

The purpose of this management response is to address the areas for program 
consideration/improvement, identified through the formative evaluation of the Understanding 
the Early Years (UEY) Initiative. The management response provides HRSDC officials with 
the opportunity to communicate their response to the key evaluation findings, to indicate 
where program policies have been modified, and to identify plans for further change. 

The formative evaluation was undertaken by HRSDC to fulfill Government of Canada 
accountability requirements, and to provide HRSDC with considerations for management 
and program policy design decisions. The evaluation examined issues of implementation 
and design, early progress in achieving immediate objectives, and issues related to 
accountability. 

The evaluation focused on the 21 UEY community projects selected for funding through 
the 2005 Call for Proposals, and was conducted approximately half way into the three- year 
funding agreements which ended in late 2008. The evaluation results are being used to 
inform the delivery of the final set of 15 UEY projects, which commenced in spring 2007 
and are scheduled to end in March 2010. The UEY Initiative sunsets in 2010-2011. 



Key Findings 

The evaluation findings indicate that the UEY design is sound and that progress towards 
full implementation in the 2005 projects was being made. For example, the Initiative’s 
mandate, objectives and intended outcomes have been consistently communicated and 
support the goals and priorities of the Department. 

The evidence also shows that the UEY model is inclusive, as well as sufficiently flexible to 
allow for the participation of a broad range of communities. Collaboration, community 
focus, credibility and communication are cited as key strengths of the Initiative. 

The evaluation identified seven considerations/improvements for the UEY Initiative that 
required action on the part of the program area, which are outlined below with the 
corresponding program response. 
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Areas for Consideration/Improvement and Actions Taken 

1. Some of UEY's long-term objectives may be too broad relative to the scope and 
size of the UEY Initiative. These objectives should be clarified or modified to 
better link them to the resources and scope of the UEY Initiative. 

The program agrees with this recommendation: 

• Departmental officials undertook a redesign of the Initiative’s Integrated 
Accountability, Audit and Risk Framework (RMAF/RBAF), including the logic 
model, performance measurement framework, evaluation strategy, risk assessment 
and audit requirements. 

• The Initiative’s long-term objectives have been modified to demonstrate a clearer 
link to resources and scope of the UEY Initiative. 

o The long-term outcomes, “increased accountability of social policies and programs 
through the incorporation of local knowledge and data ” and “enhanced support 
system within communities based on successful and sustainable community 
initiatives” have been revised to state, “ Inclusive communities that are responsive 
to the needs of children and families. ” 



2. The UEY Initiative is working to resolve process-related issues (e.g. related to 
planning and delays) and this work should continue. 

The program agrees with this recommendation. HRSDC officials have been pro-active 
in responding to process-related issues with respect to planning and delays in the 
two data collections, clarity of some data collection tools and communication between 
HRSDC and projects and will ensure that this continues: 

• Both UEY data collections, the Parent Interviews and Direct Assessments of 
Children Survey (PIDACS) and the Early Development Instrument (EDI), have been 
moved to the second year of the project life cycle to allow recipient organizations’ 
time to develop relationships within the community and to work with their 
community partners to ensure that the project’s objectives and research activities 
are understood. The inventory of programs and services has been moved to the 
first year of the project life cycle to enable the project coordinator to develop 
relationships with key service providers in the community. 

• HRSDC officials have worked with Access to Information and Privacy (ATIP) repre- 
sentatives to develop an explanatory letter and consent form for parents, to better 
describe the PIDACS data collection and obtain consent for their child to complete 
the direct assessment activities. In consultation with ATIP, a data sharing process 
has also been developed to allow recipient organizations to request their EDI and 
PIDACS project data files, for use under the conditions of the agreement. 



x 
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• Communication with project recipients has been greatly enhanced. In 2007, HRSDC 
officials met individually with the final set of UEY projects to familiarize them with 
HRSDC processes and requirements, and to enable the organizations to better plan 
their budgets and workplans. These site visits were also used as an opportunity to 
discuss the school boards’ key role in the data collections and the importance of 
transmitting information early in the project life cycle. Two information sessions 
were held in Ottawa to inform the 2007 projects about the UEY research; one session 
was held before the data collections began so that the projects would better 
understand the process and their role. 

3. Consider streamlining parent interviews. 

The program agrees with this recommendation: 

• HRSDC officials reviewed and streamlined the parent questionnaire for the PIDACS 
data collection in the 2007 UEY community projects. This process took into account 
feedback received from the 2005 UEY community projects. The length of the parent 
interview has been shortened from 90 minutes to 60 minutes on average. 

4. The linkage of research data from EDI and PIDACS may better enable research on 
possible relationships between early childhood outcomes and a number of family 
and community factors. As well, there is a need for program officials to better 
communicate with community coordinators and others in the community about the 
cohort used for the PIDACS and the usefulness of these data. The continued 
concern expressed by community representatives in this area indicates the need to 
validate the PIDACS results once they are all available. 

The program agrees with the recommendation regarding the linkage of the EDI and 
PIDACS data. HRSDC officials may incorporate this element into the design of future 
initiatives to enable research on the possible relationships between developmental 
outcomes assessed by the EDI and family and community factors examined by the 
PIDACS. 

• In the context of the current national UEY Initiative, data linkage was not part of the 
program design and both data sets are valuable to communities in their own right. 

• The EDI and PIDACS data for the 2005 projects cannot be linked because they 
were collected for two different cohorts of kindergarten children (2006 and 2007). 
The data collected for the 2007 UEY projects will not be linked as HRSDC will not 
be a recipient of the EDI data. The community projects are acquiring their 
EDI data through direct contracts with the organization responsible for the 
collection and analysis or through their provincial EDI collections. 

• Taken together, without linking, the different perspectives on young children, and their 
family life provided by the EDI and PIDACS data make the results rich. The EDI 
provides kindergarten teachers' assessments of children's outcomes in major 
developmental areas, while PIDACS makes available the results of three direct 
assessments of children’s cognitive development and parents' perspectives on their 
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children's well-being. PIDACS also provides parents' perspectives on a range of 
family characteristics and family and child experiences. 

The program agrees with the recommendation about the need for better communication 
about the cohort used for the PIDACS and its usefulness. 

• Departmental officials and research contractors met directly with representatives of 
each community project to explain that the PIDACS data, collected on a different 
cohort of children than that for the EDI, does not compromise the objectives of these 
two surveys: to find out how young children in a UEY community are faring in major 
developmental domains such as physical health, cognitive development and social 
behaviours (EDI), and to better understand characteristics of their families and 
experiences (PIDACS). 

• The community research reports produced by the PIDACS contractor based on both 
the PIDACS and EDI data, from two different cohorts of kindergarten children, 
presented a comprehensive picture of the young children growing up in a community. 
The reports were well received and useful to communities in their planning activities. 

• Departmental officials acknowledge that there may still be some degree of 
misunderstanding about the usefulness of the data given that it was collected on a 
different cohort of children. The data provided to the UEY projects are complex 
and, for most, it is the first time that communities have had access to such 
comprehensive and timely information about their young children and families. 
To support the 2007 projects, HRSDC officials hosted a two day session in January 
2009, to further explain the research and how it can be used in UEY communities. 
Departmental officials will endeavour to help the projects to use the PIDACS data 
when it becomes available, notably through data sharing agreements. 

The program agrees that the PIDACS data must be validated. 

• The PIDACS researcher validated the PIDACS data for the 2005 projects was repre- 
sentative of each community prior to writing each community’s research report. 
The PIDACS data for the 2007 projects will be validated by checking that they 
represent the kindergarten population in each UEY community project. The same 
approach will be followed for the 2007 community projects. 

5. It would be helpful to further assist communities in completing the Community 
Mapping Report. This could be accomplished in different ways (e.g. additional 
in-person training sessions, an on-line training module or a helpline). 

The program agrees with this recommendation. HRSDC officials have undertaken the 
following activities to support UEY projects to accomplish this activity: 

• Introductory mapping workshops have been delivered to representatives from both 
the 2005 and 2007 UEY projects. Training resources included participant workbooks 
and a follow-up web cast for the 2007 community projects. 
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• HRSDC officials have encouraged all UEY projects to recruit researchers who have 
some expertise using geographic information systems (GIS) software to map local 
data. As well, HRSDC officials with mapping expertise have been available to offer 
additional support. 

6. To better support the summative evaluation, the Initiative needs to consistently 
collect data at the community level on how knowledge gained within the 
UEY communities is being applied regarding child development. Guidance will 
need to be provided to the communities on how best to collect these data. 

The program agrees with this recommendation: 

• HRSDC officials have met directly with representatives from all of the UEY 
projects to provide an overview of the Initiative’s desired outcomes and key 
indicators, to provide guidance on how these data could be collected at the 
community level, and to explain the activity and final reporting templates. As well, 
UEY HRSDC performance reporting expertise are available to offer additional 
support in determining methods for collecting and reporting on identified 
performance indicators. 

• The Initiative’s intermediate outcome “increased awareness regarding child 
development” has been revised to state “families and communities understand the 
importance of children ’s development ” . Targeted indicators have also been developed 
as part of the Initiative’s revised performance measurement framework. 

7. To better support the summative evaluation, the UEY Initiative and the UEY sites 
should ensure that outcomes are linked to measurable indicators and that the 
necessary data can, and are, consistently collected. Consideration should be given to 
improving the activity report template to better facilitate the quantification and 
analysis of UEY project progress/performance. As well, there is a need for improved 
quantitative baseline data for indicators of intermediate and longer-term outcomes. 

The program agrees with this recommendation: 

• As noted previously, departmental officials undertook a redesign of the Initiative’s 
RMAF/RBAF, including the logic model, performance measurement framework, 
evaluation strategy, risk assessment and audit requirements. 

• To support the collection of baseline data on indicators to measure progress on 
intermediate and longer-term outcomes, HRSDC officials have developed a reporting 
system to ensure that projects’ progress and reported results are being collected and 
linked to measurable indicators. 

• To better facilitate the quantification and analysis of UEY project progress and 
performance, HRSDC officials have revised both the quarterly activity and final 
report templates that are completed by each project. 
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o Two interactive one-day workshops were held in 2008, to inform UEY project 
representatives about HRSDC’s approach to performance measurement and 
evaluation, to ensure that project reports meet HRSDC requirements and at the 
same time are beneficial for the recipient organizations. 

• Departmental officials will be working with the Evaluation Directorate, to finalize 
the design of the summative evaluation and the quantitative indicators that will be 
employed in the summative evaluation to measure UEY’s intermediate and long-term 
efforts or impacts. 



Conclusion 

The conclusions outlined in the formative evaluation include several positive findings as 
well as areas for improvement. These observations will provide sound advice for senior 
management and they have helped to inform implementation and management of the final 
set of 15 UEY projects which commenced in 2007. Changes have been made to the 
Initiative since the study period to address these observations, and HRSDC officials are 
committed to continuous improvement to the Initiative. 



XIV 
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1. Program Overview and 
Evaluation Context 



1 .1 Purpose and Organization of the Report 

This report presents the findings from the formative evaluation of the Understanding 
the Early Years (UEY) Initiative. The formative evaluation was undertaken to assess the 
development of the Initiative by examining issues of design and implementation, early 
progress in achieving results (i.e. progress towards achieving the immediate objectives), 
and issues related to accountability. This report also provides some preliminary 
guidance/suggestions on the design of a summative evaluation of the UEY Initiative. 
The information provided in this report consists of six sections: 

• Section 1 provides an overview of the UEY Initiative, plus the evaluation context 
and methodology; 

• Section 2 examines issues of design and implementation; 

• Section 3 examines progress toward immediate objectives; 

• Section 4 examines issues related to accountability; 

• Section 5 identifies some suggestions for the design of a summative evaluation; 

• Section 6 presents the main conclusions 



1 .2 Overview of the UEY Initiative 



1.2. 1 Program Description 

The UEY Initiative helps communities gather and use local information about their 
children to: 

• identify gaps in services and programs for young children and their families; and 

• foster partnerships among community groups and individuals to make informed decisions 
about the best programs and services for young children to thrive. 

The UEY Initiative was announced in 2004 as a seven-year national initiative (2004-05 to 
2010-11) to support up to 100 communities across Canada. The Initiative built on the success 
and lessons learned in 12 UEY pilot projects, which began in 1999 and ended in 2007. 
The objective of the pilot projects was to enhance knowledge about community factors that 
influenced the early development of children. 
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The 2004 Budget set aside $68 million for the UEY Initiative over seven years ($34.5 million 
in operating funds, and $33.5 million in contribution funding). The Expenditure Review of 
September 2006 resulted in a reduction in the Initiative’s original budget to $45.3 million 
($23.9 million for operating, and $21.4 million for contributions). 

The UEY Initiative provides three years of contribution funding to sponsoring organizations 
which act in concert with their communities to generate local information (as discussed in 
Section 1.2.3 and Section 1.2.4). The information allows members of funded communities to 
work together to identify gaps in services and programs for young children and their families, 
and to develop a Community Action Plan to guide community decisions on how best to 
support families with young children. The Initiative includes rural and urban areas and 
promotes the participation of communities with children from diverse backgrounds including 
First Nations, Inuit and Metis children, children in immigrant fa mi lies and official language 
minority communities, and children in low-income fa mi lies. 



1.2.2 UEY Design and Delivery 

The UEY Initiative is delivered through National Headquarters of Human Resources and 
Skills Development Canada (HRSDC) and is managed by the Community Development 
and Partnerships Directorate. The UEY team consists of project analysts who oversee the 
contribution agreements held by sponsoring organizations in the UEY communities, and 
research staff who are responsible for the management of two large data collections 
and related activities. Service Canada officials provide assistance with outreach activities 
and are part of the internal review of proposals submitted as part of the UEY community 
selection process. 

UEY communities are selected through a Call for Proposals process. The submitted proposals 
are first screened to determine whether they meet the Initiative’s mandatory requirements. 

• The applicant organization must be a legally incorporated 3 not-for-profit organization 
and be actively pursuing social development issues. 

• The proposed community must be located within a certain geographical location defined 
by boundaries understood by residents. The geography must be continuous or contiguous. 
People within these boundaries share a sense of belonging and identify with all, or parts, 
of the geographic community. 4 This requirement is designed to encourage ownership 
and community involvement around the children’s early development. 



3 Although it was not mandatory that the applicant organization be legally incorporated in the 2005 Call for Proposals 
(the focus of this formative evaluation), this was a mandatory requirement in the 2006 Call for Proposals. 

4 A UEY community may be large or small, urban or rural, and may be what is commonly understood as a 
neighbourhood within a larger community. Alternatively, it may be a large region where there are many towns 
linked by geography. Residents must identify with the community, so that UEY participants have a sense of 
ownership and community involvement around the issue of children’s readiness-to-learn. Depending on the focus, a 
UEY site may also be a sub-set of a geographical community that only looks at children sharing a common 
characteristic (e.g. language). Applicants are required to provide a map of their UEY community, including the 
boundaries that define the community. 
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• The proposed community must have an existing community coalition with experience 
and a record of accomplishments in dealing with social issues. 5 

• Applicants are required to identify (in advance) all participating schools/school boards 
(or their equivalent) willing to participate, 6 because school boards play a key role in 
facilitating the collection of some of the local data. 

• Within the community, there must be suitable potential candidates (with appropriate 
skill sets and leadership qualities) for the UEY position of community coordinator. 

• The proposed community must have at least 300 five-year old children entering senior 

n 

kindergarten (or equivalent) in the upcoming school year. 

Proposals that meet the mandatory requirements undergo an internal assessment by National 
Headquarters of HRSDC and regional staff plus an external assessment by provincial 
government representatives. Following the internal and external assessment process and 
Ministerial approval, HRSDC enters into a three-year contribution agreement with each 
not-for-profit organization that has a successful application on behalf of its community. 
Each of these organizations becomes the project sponsor for the UEY community. 

With the UEY funding, the sponsoring organization hires a community coordinator to 
manage all aspects of UEY. The funding may also be used to hire a community researcher 
to carry out the research on community programs and services and to interpret and map the 
research results. The sponsoring organization and community coordinator work with a 
UEY community coalition (which consists of parents, educators, service providers and 
others interested in the well-being of children). Together, they engage the community 
around the importance of early childhood development and the development of a 
Community Action Plan (as discussed in Section 2.1.3, Section 2.1.4 and Section 2.1.5). 
Also, the UEY community staff organize and coordinate information sharing and 
dissemination activities to promote UEY activities within their community and strengthen 
partnerships among the various stakeholders (e.g. parents, schools, service providers and 
the business sector) to address the needs of young children. 

In the fall of 2005, 21 community projects across the country were selected to be UEY sites 
(see Appendix A). These projects are the subject of this evaluation. Under the three-year 
UEY contribution agreements, the UEY sites received on average between $350,000 and 
$375,000 in funding for their community project. As well, UEY communities received 
training and technical support (e.g. orientation meeting, community mapping workshop, 



5 The group does not have to refer to itself as a coalition, and the group may take a range of forms. Membership 
should be broad and include parents, teachers, school trustees, social service providers, health service providers, and 
representatives of other civic services. An existing coalition that does not include key groups (e.g. schools/school 
boards and program and service providers) must recruit those groups for the UEY project. 

6 Since the UEY site may be a sub-set of a geographical community and may be looking at only some of the children 
within that area (e.g. only children entering French language senior kindergarten), the UEY project may involve the 
participation of only some of the schools or boards within an area. The proposal for funding must include letters of 
support from each participating school board involved in the UEY project indicating the willingness to provide lists 
of the children and contact information for their families, and their willingness to facilitate the collection of 
UEY data in their schools. 

7 These are children who will start school at age five and be five or six during the data collection period. 
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EDI workshop). A second group of 15 communities is being added to the Initiative from 
the 2006 Call for Proposals. 



1.2.3 UEY Research 

UEY research generates several types of local information that is made available to each 
UEY project. The local information includes parents’ and teachers’ perspectives on the 
development of kindergarten children, direct assessments of kindergarten children’s cognitive 
abilities, parents’ perspectives on family circumstances and children’s experiences, information 
on local programs and services, and local socio-economic characteristics. Each of these 
types of information is discussed in more detail below. 



Teachers Complete the Early Development Instrument (EDI) 

Kindergarten teachers complete the Early Development Instrument (EDI) for all children 

in senior kindergarten (or the equivalent) in schools participating in the UEY project. 

• The EDI is a population-based measure used to assess the development of kindergarten 
children as a group. It provides teacher ratings of children’s behaviour and capacities in 
five areas: physical health and well being, emotional maturity, social competence, 
language and cognition, and communication and general knowledge. 

• The EDI was developed under the direction of Dr. M. Janus and Dr. Dan Offord at the 
Offord Centre for Child Studies at McMaster University. Under contract to HRSDC, 
the Offord Centre was responsible for the EDI assessments for the 21 UEY projects, 
including site-specific reports for each community. 



Parent Interviews and Direct Assessments of Children Survey 
(PIDACS) 

The Parent Interviews and Direct Assessments of Children Survey (PIDACS) has a number 
of components. The parent interview component provides information on family background 
and family processes, including socio-economic status, parenting style, and use of 
community resources. Parents also report on their kindergarten child’s development and 
experiences, and activities at home and in the community. Direct child assessments 
components examine receptive vocabulary, early literacy and general cognitive skills, and 
early numeracy. 

• The instruments used by the PIDACS are based on those used by the National Longitudinal 
Survey of Children and Youth (NLSCY) since 2000. The instruments are administered 
to a sample of kindergarten children in each community and the person most 
knowledgeable about each child (usually a parent). 
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• Under contract to HRSDC, R.A. Malatest and Associates was responsible for the PIDACS 
data collection in the 21 UEY projects, including a Community Research Report for each 
community. 

• A standard letter of information and consent form for parents was provided by HRSDC 
and used in each UEY community (with some local information added if applicable), 
to ensure that parents were adequately informed about the PIDACS research and 
consented to their local UEY project receiving a data file for use on behalf of young 
children in the community. 



Inventory of Programs and Services 

Each UEY project develops its own inventory of local programs and services available to 
young children and their families. Relevant programs and services include health clinics, 
schools, family resource centres, recreational facilities (including parks and pools), and 
child care facilities. 



Socio-economic Data 

Each project also compiles data on community socio-economic characteristics such as 
levels of employment, education and home ownership. This information is usually based 
on neighbourhood- specific data from the latest Census of Canada, but may also include 
local administrative information. HRSDC has provided 2001 Census data to sponsoring 
organizations on request through its licensing agreement with Statistics Canada. 



1.2.4 Using and Disseminating Information 

The Initiative’s objectives include increasing the communities’ understanding of how 
young children in the community are doing and how the community is supporting those 
children. Certain reports are developed by each UEY community to help use and share 
the results from its UEY research. 

• The Community Research Report presents findings on the development of kindergarten 
children as provided by their parents, teachers and the children. This report also presents 
information about families and the community provided by the parents. Each community- 
specific research report is developed by the research firm responsible for the PIDACS data 
collection. 

• The sponsoring organization in each funded community creates a Community Mapping 
Report to illustrate the UEY research findings for community members. The report 
includes maps that provide a “picture” of the relationships between community factors 
(e.g. location of parks, libraries and child care centres), community socio-economic 
characteristics (e.g. rates of employment and home ownership), and children's develop- 
mental outcomes (e.g. social competence). This information comes from the inventory of 
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programs and services, neighbourhood-level Census data, and teacher data on children’s 
development. 

• A Community Action Plan is developed by the sponsoring organization in collaboration 
with community organizations and individuals. The Action Plan is based on the findings 
from the UEY research. It identifies measures to improve services and programs for 
young children. It also identifies measures to encourage community members to 
work together to address the needs of young children. As well, it proposes ways 
to address gaps in community supports for children and serves as the community’s 
blueprint for action. 

The process of developing the Community Action Plan and the events and activities to 
disseminate the research information (to parents, service providers, educators and others) 
are used by the UEY sponsoring organization and community coalition to engage the 
community around ways to enhance the importance of early childhood development. 



1.2.5 Key Players in a UEY Project 

In addition to HRSDC, a variety of key players are involved in implementing a UEY project. 

• The community sponsor/sponsoring organization is responsible for the management 
and delivery of the project (as the responsible agent in the contribution agreement with 
HRSDC). The sponsoring organization has overall responsibility for project deliver- 
ables and results. It also has overall responsibility for providing financial and 
activity/progress reports and a final report to HRSDC. As well, it is also responsible for 
hiring the community coordinator and the community researcher (if one is hired). 

• The community coordinator manages all aspects of UEY-related activities within the 
UEY site and is responsible for working with the researchers (internal and independent) 
and with the community coalition. The community coordinator reports to the 
sponsor/sponsoring organization and is responsible for the completion of project 
deliverables, relationship building, and developing community interest in UEY activities 
and the importance of early childhood development. The coordinator works with the 
sponsoring organization, community researcher and community coalition to develop the 
evidence-based Community Action Plan, and is responsible for facilitating data 
collection, local research, and information sharing. The community coordinator also 
trains the teachers to participate in the EDI. 

o 

• The community researcher compiles the Inventory of Community Programs and 
Services, interprets data, liaises with data collection contractors on reports, and produces 
the Community Mapping Report. 

• The community coalition has an advisory role. It works with the community 
coordinator and sponsoring organization to understand the UEY data, engage the 
community, and develop the Community Action Plan. 



Not all sites have a researcher. In some communities the community coordinator carried out this function. 
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• The independent contractors collect the EDI and PIDACS data. They also produce 
the EDI reports, Community Research Reports, and the EDI and PIDACS data files. 

• Provincial and territorial officials , while not key players in a UEY project, participate 
in the external review of proposals and are involved in some community coalitions. 



1.3 Evaluation Scope and Methodology 

The formative evaluation was focused on examining the development of the UEY Initiative 
by examining issues of implementation and design, progress in achieving immediate 
outcomes, 9 and issues related to accountability. In addition, the evaluation team was asked 
to provide preliminary guidance on the design of the summative evaluation. 

The formative evaluation focused specifically on the 21 communities selected for 
UEY funding in 2005 (i.e., the 2005 round of UEY sites). It was conducted approxi- 
mately half way into the three-year funding agreements with these communities. 



1.3.1 E valua tion Approach 

The approach developed for the formative evaluation was based on the use of multiple 

lines of evidence drawn from three main sources. 

• The review of documents and files included existing UEY documents and the 
administrative files for each of the 21 communities selected in 2005. The majority of 
reviewed documents were provided by HRSDC and included: background on the 
UEY Initiative, information on UEY data collection and support to sites, community 
proposals, contribution agreements, evaluation and results-based reports, and adminis- 
trative files. Also included was the Early Implementation Review produced by HRSDC 
Internal Audit. 10 

• A literature review was conducted to assess how research and evaluations of similar 
programs could inform the UEY summative evaluation. As well, the literature review 
examined similar community-based programs in Canada and other countries to help 
identify ways to augment the UEY design, data collection and analysis methods. 
Also included was a review of recent research literature on the role of communities in 
fostering early childhood development. 11 



9 Immediate outcomes are anticipated within one year of implementation (as discussed in Section 2.1). 

1(1 HRSDC’s Internal Audit Services completed an Early Implementation Review of the UEY Initiative in 2006 to 
determine whether management practices and systems are developed and aligned to support program objectives, and 
are operating as intended. 

11 This involved conducting a database search for relevant research publications (e.g. Web of Science Citation Index, 
Psychlnfo, PubMed, and Sociology). 
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• In-depth interviews with key informants (n=44) were conducted in the spring and 
summer of 2007 (approximately half way into the three-year funding agreements with 
the 21 UEY communities) with representatives of the following groups: 

o UEY program management and staff (n=6); 

o independent contractors, including persons from firms working under contract with 
HRSDC who are involved in data collection and analysis (n=3); 

o an Access to Information and Privacy expert with knowledge of the UEY Initiative 
(n=l); 12 

o UEY community representatives (n=24) (i.e. UEY community sponsors/sponsoring 
organizations (n=9), UEY community coordinators (n=9), and UEY community 
coalition members (n=6)); 

o community applicants whose UEY application was not funded (n=5); and 

o representatives from provincial and territorial government departments, and experts 
in the field of community influences on childhood development (n=5). 

1 T 

Ten of the 21 UEY communities were selected to be part of the key informant interviews. 
These communities were selected to ensure representation of an urban/rural, large/small, 
region and ethnic mix of the population. All key informant interviews were conducted 
by telephone. 14 

To provide relative weight to the findings presented throughout the evaluation report, a set 
of guidelines were used to describe how key informants responded to a given question: 

• None: no key informants expressed a particular view. 

• A few: one or two key informants have a particular view. 

• Some: between one-quarter and half of the key informants hold the same view. 

• Most: a majority (more than half) but not all key informants holds the same view. 

• All: all key informants expressed the same/similar views. 



12 This individual was interviewed using the same interview guide as the independent contractors. 

13 Although the original evaluation methodology called for a total of 50 interviews to be completed across all 21 sites, 
findings were required for the TB submission process. Therefore, due to timing issues, key informant interviews 
were conducted with representatives of 10 of the 21 UEY sites. 

14 Contact information for the UEY community sponsors was provided by UEY program management. Contact information 
for coalition members and community coordinators was obtained from the UEY community sponsors. 
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1.3.2 Methodological Challenges and Limitations 

Although the combined use of these three sources of information provides a useful 

information base for the formative evaluation, certain limitations should be noted. 

• Although some data on outcomes was drawn from the file review process, these data 
should be treated with caution because some definitional challenges were encountered 
in aggregating the data across projects. For example, the reporting of quantitative 
data/information by UEY sites is, to some extent, open to each site’s interpretation of 
the indicators specified by the UEY Initiative. As well, problems in determining what 
constituted a partnership made it difficult to interpret community partnership data. 

• Due to some turnover among staff in the UEY communities, it was sometimes difficult 
to reach the most suitable key informants. 

• To ensure that one opinion did not unduly influence the findings, a ‘representative 
approach’ was used to quantify the key informant results. A limitation of this approach, 
however, is that it can minimize the views of individuals who may have unique 
knowledge of an issue (based on their specified area of expertise). Therefore the 
evaluation team made every effort to ensure that this type of unique knowledge was 
factored into the evaluation results. 

• Another potential limitation is that key informant interviews were conducted with 
representatives of only ten of the 21 UEY sites. The effects of this limitation were 
minimized, however, because the majority of data collection focused on all 21 UEY sites. 
For example, program files were reviewed from each of the 21 sites. As well, the other 
categories of key informants provided a broader perspective of the UEY Initiative. 
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2. Findings on Design and Implementation 



This section examines issues in the following areas: 

• UEY objectives; 

• flexibility and project selection; 

• roles and responsibilities of UEY players; 

• the strengths, challenges and suggested improvements in the case of UEY design and 
key research tools; and 

• lessons learned (specifically, what has worked well for the 2005 round of UEY sites, 
where are changes needed, and what refinements have been made). 



2.1 UEY Objectives 

This part examines whether UEY objectives are clear and achievable. It also examines whether 
the 21 UEY sites have experienced any difficulties in understanding or making progress 
towards their project’ s immediate objectives and expected results for the UEY Initiative. 

2. 1. 1 Are UEY Objectives Clear and Achievable ? 

UEY program documentation indicates that the broad objectives of the UEY Initiative 
are: to strengthen the ability of communities to use quality local data to help them make 
decisions to enhance children’s lives; and to enable community members to work 
together to address the needs of children. The intended program outcomes (objectives) 
are shown in Table 2.1. 



Table 2.1 

Summary of Intended Program Outcomes 


Type of Outcome 


Intended Outcome 


Immediate outcomes 
(objectives) anticipated 
within one year of 
implementation 


• increased support for communities 

• increased availability and sharing of information on the role of families 
and communities in child development 

• increase awareness and use of parent and community knowledge 
regarding child development 


Intermediate outcomes 
(objectives) anticipated 
within two to three years 
of implementation 


• increased community ability to form partnerships across sectors to 
respond to child development issues 

• increased ability of communities to help parents make choices for their 
children based on local knowledge and data 


Longer-term outcomes 
(objectives) anticipated 
in four or more years 


• enhanced support system within communities based on successful 
and sustainable community initiatives 

• increased accountability of social policies and programs through the 
incorporation of local knowledge and data 


Ultimate goal of the 
UEY Initiative 


• strengthen Canadian families with children by providing choice and 
support that recognizes the diversity of their needs 


Source: Document review. 
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Finding: The mandate, objectives and intended outcomes of the UEY Initiative are 
consistently communicated. 

Findings from the Early Implementation Review concluded that the various elements of 
strategic planning for the UEY Initiative - including mandate, objectives, and intended 
outcomes - support the vision, goals and priorities of the department. Files and documents 
reviewed for the formative evaluation supported the conclusion that the mandate, 
objectives and intended outcomes of the UEY Initiative are all consistently communicated. 

Finding: The focus and measurability of some of the Initiative’s intended outcomes 
could be improved, particularly in the case of the longer-term and ultimate outcomes. 

Program officials interviewed as key informants had mixed opinions on whether the 
Initiative’s outcomes (immediate, intermediate and longer-term) are well-defined and 
achievable. 

• A few key informants felt that the longer-term outcome of “increased accountability of 
social policies and programs through the incorporation of local knowledge and data ” 
is difficult to attribute to UEY, and therefore UEY cannot be expected to have a 
measurable impact in this area. 

• Similarly, a few key informants noted that the ultimate goal of “strengthened Canadian 
families with children by providing choice and support that recognizes the diversity of 
their needs ” is very broad and should be more focused. 

It should be noted that these comments related to improvements/refinements rather than a 
redesign of the Initiative. They are also consistent with revisions of the UEY logic model 
underway at the time of the formative evaluation. 



2.1.2 Participants ’ Understanding and Early Experience 

Finding: Participants appear to have a clear understanding of their project’s 
immediate objectives and expected results for the UEY Initiative. 

Key informant interviews with community representatives (i.e. community sponsors, 
coordinators and coalition members) and program officials provided evidence that each 
of the UEY sites has a clear understanding of the community’s immediate and expected 
results for the UEY Initiative. All community key informants were able to provide a 
description of their project’s objectives, which were reflective of the outcomes of the 
UEY Initiative. 

Finding: The evidence indicated that all 21 UEY communities have made progress in 
implementing their project and were making progress towards their immediate 
objectives and expected results for the UEY Initiative. 

The file review found that all 21 communities have made progress in implementing their 
UEY project and were making progress towards the immediate objectives and expected 
results. For example, at the time of the evaluation, all communities had completed their 
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EDI data collection and had received the EDI results report from the independent 
contractor. All sites had developed information sharing products on UEY for their 
community, most communities had started or completed their Inventory of Programs and 
Services, and over half had initiated activities for the preparation of their Community 
Mapping Report. 

Most key informants who were program officials believed that UEY communities should 
be able to make good progress on achieving their expected results. At the same time, 
however, some key informants noted that certain communities had made more progress 
than others. 

Finding: Most sites experienced some early difficulties, which were largely the result of 
challenges within individual communities (e.g. buy-in challenges and staffing) rather 
than due to flaws in UEY design. 

Although the 2005 round of UEY sites were found to be on-track to achieve their immediate 
outcomes, there was evidence that most sites experienced some early difficulties. These 
difficulties reflected the realities/challenges within each community rather than design or 
delivery problems with the Initiative itself. The following early challenges were identified by 
key informants and supported by the document and file review. 

• Geographic and/or demographic factors: some communities faced challenges associated 
with the composition of the population or their geographic location. For example, evidence 
shows that, due to language and cultural differences, incorporating immigrant and 
Aboriginal groups into the project can be challenging. Some sites reported that travel time 
and distances between communities posed a challenge for project implementation. 

• Collaboration of schools /school boards: a few projects experienced difficulties getting 
buy-in on the part of schools and/or school boards. Sometimes the schools were 
involved in a number of projects not directly related to the UEY Initiative, which 
resulted in overload and confusion for teachers. In other cases, the UEY project 
involved a number of schools with no single governing board. This meant community 
sponsors had to work through levels of bureaucracy or decision-makers for each 
school. In a few communities, schools or school boards with ethnic or multicultural 
populations were reluctant to participate in data collection processes (EDI and 
PIDACS). They did, however, participate in activities stemming from the initiative 
(e.g. school breakfast programs). Further details on the participation of schools/school 
boards are presented in Section 2.4.2 and Section 2.4.3. 

• Community buy-in: some community representatives noted that it was initially 
difficult to get community members and organizations to buy into the project when 
they did not understand the purpose or potential benefits of the UEY initiative. 
Engaging various types of stakeholders, such as profit and non-profit organizations and 
local politicians, was also identified as an early challenge. 
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• Staffing: a few community representatives mentioned staffing challenges within their 
communities, such as staff turnover or finding the appropriate individuals with the 
necessary knowledge and skills (i.e. to be a community coordinator or community 
researcher). Some identified staff turnover at HRSDC as a challenge for communities 
because new HRSDC project analysts needed time to become familiar with UEY projects. 
Although some staff turnover should be anticipated, given the three-year duration of UEY 
projects, UEY management has been working to minimize the impact of HRSDC staff 
changes on communities. 

• Delays: were often cited as early challenges. Frequent examples included delays in 
collecting, receiving or sharing data. Other delays occurred because of difficulties 
in obtaining the collaboration of school boards, staffing challenges within the community 
(e.g. in hiring a community coordinator), or obtaining community buy-in. In some cases, 
the start of the project was stalled by a delay between the application stage and funding. 
This had further consequences when staff turnover during the waiting period resulted in a 
need to obtain replacements (causing additional delays). There was also a substantial 
delay in getting the PIDACS data collection started (as discussed Section 2.4.1 and 
Section 2.4.3). 

It should be noted that the document and file review found that the selection 
process/Call for Proposals specifically screens for factors that are expected to strongly 
contribute to a community’s ability to successfully meet its UEY project objectives. 
The document and file review also revealed that the UEY’s Call for Proposals process 
encourages communities to anticipate and resolve some of these challenges at the 
application stage. For example, communities are required to plan and clearly identify 
the geographic boundaries of their UEY project (as noted in Section 1.2.2). Similarly, 
communities are required to provide evidence of community and school board buy-in 
and participation in the project. The experience of the 2005 round of UEY sites 
suggests that it would be useful to give additional consideration to identifying ways to 
alert potential applicants to the various types of early challenges and to inform them of 
solutions that have worked well in other UEY communities. 



2.2 Flexibility and Project Selection 

This section examines whether the UEY model is flexible in accommodating differences 
across communities and whether it is inclusive of diverse communities. This section also 
examines whether the proposal review process included adequate regional representation. 
It also examines the experience of non-funded applicants. 
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2.2. 1 Flexibility and Inclusiveness of the Model 

Finding: The UEY model is inclusive of the broad range of communities and is 
sufficiently flexible to accommodate their participation. Inclusiveness is balanced, 
however, against the Initiative’s mandatory requirements ( which help to ensure that 
communities receiving UEY funding are able to successfully meet their objectives ). 

The file and document review indicated that the UEY Initiative is committed to providing 
funding to a broad spectrum of communities. The Call for Proposals was widely promoted 
across Canada, and communities had many opportunities to learn about the Initiative and 
submit a proposal. In addition, the criteria for funding encouraged the participation of a 
diverse range of communities. As well, proposals were screened with a view to achieving 
regional representation as much as possible, to ensure that urban and rural communities 
were among the successful candidates. Effort was also made to include children from 
marginalized or disadvantaged communities in the selected projects. 

The profile of the 21 sites selected for the 2005 round of UEY funding indicates that these 
communities are very diverse in terms of their population density, socio-economic status, 
ethno-cultural composition, languages and number of new Canadians. Eight of the 21 sites 
are urban, six are rural and the other seven are mixed. Population information available for 
16 of the 21 sites indicate that their populations range from 12,000 to 774,072 people. 
The regional distribution of the 21 sites is as follows: Ontario (seven sites); British Columbia 
(six sites); Nova Scotia (three sites); Quebec (two sites); and Saskatchewan, Manitoba and 
New Brunswick (one site each). 

Program officials who were key informants were unanimous in their view that UEY is 
inclusive of diverse communities across the country, and most provincial/territorial govern- 
ment representatives and experts agreed. Community members who were key informants 
strongly supported the perceptions of program officials. For example, while some 
UEY communities, particularly rural sites, are described as being quite homogeneous, others 
are described as quite culturally and socially diverse. Similarly, some communities have a 
significant proportion of low-income families or single parent families, while others are quite 
affluent overall. 

Most key informants from funded communities and key informants from non-funded 
applicants (i.e. communities that applied for, but did not receive UEY funding) agreed 
that the UEY model is inclusive of children and families of diverse backgrounds and 
circumstances. They illustrated this by describing the groups included in the Initiative 
(e.g. Aboriginal, Francophone, ethnic/minority, rural, lower income). Some community key 
informants also described a broad range of efforts undertaken by their community to ensure 
inclusiveness such as translation of data collection tools, using ethnic or multicultural media 
(i.e. not English or French) for communication, and conducting parent interviews in 
languages other than English or French. 
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Despite evidence of diversity, the Early Implementation Review (2007) identified some 
gaps in regional coverage in the 2005 Call for Proposals. The formative evaluation found 
that some program officials and provincial/territorial government representatives felt that 
more vulnerable communities and Aboriginal communities may be excluded because 
of not meeting the mandatory requirements. 15 This concern was also noted by some 
non-funded applicants (see Section 2.2.3). These key informants acknowledged, however, 
that communities not meeting the mandatory requirements of the Call for Proposals are 
unlikely to be able to successfully implement a UEY project. 



2.2.2 Adequacy of Regional Participation in 
Selection Process 

Finding: The proposal review process was generally considered to work well. 
The process was seen to adequately include the participation of regional representatives 
and experts, and to incorporate their feedback fairly and appropriately. 

The Early Implementation Review concluded that the assessments conducted by 
provincial/territorial government representatives and regional early childhood development 
experts are intended to ensure objectivity in decision-making and minimize the risk of 
error, abuse or fraud. 

The document review conducted for the formative evaluation indicated that the proposal 
review process used to select the 2005 round of UEY sites included internal and the 
external assessments performed by provincial government officials, Department staff and 
regional representatives. 16 During the assessment process, regional representatives of 
Service Canada and provincial/territorial government representatives contributed their 
knowledge of regional and local non-profit organizations that applied for UEY funding. 
Upon completion of the assessments, each proposal was assigned an overall weighted 
score (using a 60 percent weighting for internal assessments and a 40 percent weighting 
for external assessments). Finally, environmental and risk assessments of each proposal 
were conducted, and the recommendation rationale was prepared. 

Both the proposal review process and the participation of regional representatives are viewed 
positively. Key informants familiar with the process (i.e. program officials, provincial/territorial 
government representatives and experts in early childhood development) noted there was a 
good balance between UEY staff and provincial/territorial staff, with each providing input 
within their areas of expertise. They also suggested that changes to the assessor review 
guide have made the process more user-friendly, have reduced duplication and are well 
aligned with the proposal. They also noted that the template for the proposal assessment is 
well thought out, and results in a better proposal and a more efficient review process. 



15 For example, they might not have a community coalition in place, might be small in size or spread over a large area. 
1(1 The internal assessment involved two lead assessors reviewing the proposal and leading the group discussion to 
conduct the assessment. The external assessment was completed by two assessors (provincial or territorial staff 
and/or expert in early child development). 
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Finding: Although participation of regional representatives in the proposal review 
process was viewed positively, a lack of information on the overall context and the 
potential for workload/scheduling problems were identified as two challenges in the 
external assessment process. 

While there is little evidence of challenges associated with the internal proposal assessment 
process, provincial/territorial government representatives and experts identified two key 
challenges with the external assessment process. 

• Lack of information on context : each province/territory conducts the assessments in 
isolation. As a result, reviewers do not have a good sense of the balance (in terms of 
communities selected) across the country. For example, reviewers indicated that they 
did not know how many UEY sites in total would be approved for each province and, 
therefore, did not know how many proposals to recommend for funding by province or 
territory. 

• Workload/scheduling', external reviewers participate in the proposal review process in 
addition to their regular duties. A few key informants noted that this adds to their usual 
workload and can result in scheduling difficulties, although they acknowledged that the 
UEY Initiative has little control in this area. 



2.2. 3 Experience of Non-Funded Applicants 

Finding: The experience of noil-funded applicants was mixed, reflecting the unique 
situation and challenges faced by each community. Their more detailed feedback 
suggested that the mandatory requirements ( which are intended to ensure the capacity 
of the community to be involved in the Initiative and the integrity of the data ) were not 
consistently well understood. 

Based on their experience with the UEY application process, key informants who were 
non-funded applicants (n=5) were asked to comment on what they felt worked well. Most 
explained that the UEY proposal process helped them to bring the community together 
and, as a result, they were able to form networks that continue to do work on early 
childhood issues (although without funding from the UEY Initiative). Some key 
informants also reported that the networks they formed would help them to develop ideas 
and to submit a stronger proposal next time. 

The non-funded applicants were also asked what lessons they learned about the UEY 
Initiative, based on their own unique experience during the application process. 

• One key informant felt that there was a lack of information available during the 
application process, leaving little opportunity to address potential shortcomings in the 
community’s proposal. Another community representative, however, offered a different 
perspective and explained how feedback from UEY officials during the process allowed 
the community to adjust the proposal and include a stronger role for the school board 
and, as a result, submit a successful (second) application. 



Formative Evaluation of the Understanding the Early Years Initiative 




• Another interviewee explained that, during the application process they discovered that 
they had misinterpreted the purpose of the UEY Initiative. They withdrew their proposal 
when they realized that they had already completed a similar initiative in their 
community. 

• Another key informant explained that during the application process they came to the 
conclusion that the UEY Initiative was better suited to large urban communities and 
concluded that their community was too small to qualify as a UEY site. 

Non-funded applicants were also asked to comment on what they felt did not work, and 

needed to be changed. 

• Mandatory requirements : most had concerns about the requirements they had to meet 
in the application process (i.e. most would like the Initiative to be more inclusive). 
For example, one key informant explained how the community chose an umbrella 
organization as a sponsoring agency, but was rejected because the sponsoring 
organization had not been in operation for the required minimum of three years. 
Similarly, another key informant talked about how the criteria excludes small 
communities (e.g. with less than 300 kindergarten children) and needs to be revised to 
be relevant to First Nations communities. 

• Explaining their community: one key informant felt the application template did not 
allow for a full explanation (when more details might have helped at the review stage). 
Another key informant found it difficult to explain how their community worked using 
the application template, and felt HRSDC officials should be more flexible. 



2.3 Roles and Responsibilities 

This section examines whether the roles and responsibilities of key players in the UEY 
Initiative are clearly understood and being implemented as intended. 

Finding: Evidence indicates that roles and responsibilities of the key players were well 
understood at the time of the evaluation and were implemented as intended, although 
there had been some clarification in response to initial challenges. 

The specific roles and responsibilities of the key players were described Section 1.2.5. 
According to UEY program officials and program documentation, the roles and 
responsibilities of key players were explained and clarified through UEY orientation 
sessions and through written communication. Most key informants who were community 
representatives, program officials, and independent contractors felt that roles and 
responsibilities were well understood at the time of the evaluation. As with other aspects 
of UEY, there was some variation across communities in how the roles and 
responsibilities were implemented. However, that variation remained within the 
parameters of the UEY Initiative. 
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The document and file review and key informants indicated some initial challenges were 
experienced (generally site-specific and reflected the variations across the UEY sites). 
The evidence also indicated that those challenges have been largely overcome. The initial 
challenges are listed below for each key player. 

• In a few cases there was a need to clarify HRSDC’s expectations with respect to the level 
of involvement of the community sponsors/sponsoring organizations. A program 
official key informant noted that this clarification was done during the process of 
implementing the contribution agreement in each community, which resolved any early 
confusion. 

• There was an early need to clarify the role of community coordinators with respect to 
their relationship to the independent contractors (i.e. that the coordinators do not work 
for the independent contractors). Independent contractors who were key informants 
noted that some community coordinators had stronger working relationships within the 
community than others and were able to leverage their relationships and play a stronger 
leadership role in their community. 

• In a few cases there was a need to clarify that the community coalition had an advisory 
role and not a lead role in the community’s project. As well, a few community 
coordinators who were key informants noted an appreciable decline in the numbers of 
coalition/community members attending meetings over the course of the project. 
Independent contractors noted that there was variation across communities, with some 
community coalitions playing a larger role than others in the UEY projects. 

• Not all of the 21 UEY sites hired a community researcher. In communities where a 
community researcher was hired, there was no evidence of a lack of understanding or 
clarity with respect to the role of the researchers. 

• There was an early need to clarify the roles and responsibilities of the independent 
contractors in the EDI and PIDACS data collection and reporting process. Sponsors 
and coordinators from a few communities reported a misunderstanding on the part of 
independent contractors regarding what was required of them. There were a few cases 
of problems resulting from miscommunication between the independent contractors 
and the school boards, but one key informant noted that these cases could have been 
due to staff/management changes at the boards. There was also evidence of 
coordination challenges, and a few communities felt the independent contractors 
expected an unreasonable level of support from the community. 

Finding: Opinions were mixed on the clarity and implementation of the Department’s 
role, largely due to staff turnover at HRSDC. 

While some sponsors and coordinators reported having positive experiences with HRSDC, 
others reported significant problems related to staff turnover and delays. According to some 
community representatives, staff turnover at HRSDC meant that new project analysts had 
to “catch up”, which led to communication difficulties between HRSDC and the 
communities. At the same time, however, other community representatives praised the 
professionalism and assistance provided by HRSDC project analysts. The independent 
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contractors also had mixed opinions of the Department, and one also noting the challenges 
to the communities associated with staff turnover at HRSDC. 



2.4 UEY Design and Key Research Tools 

This section begins by examining the key strengths and major challenges/limitations of the 
design of the UEY Initiative. It also identifies some suggestions for improving 
UEY design. The rest of the section presents the findings on the strengths, 
limitations/challenges and suggested improvements for the EDI, the PIDACS and the 
Community Mapping Report. Areas where changes/refinements had already been made at 
the time of the evaluation are discussed in Section 2.5.2. 



2.4. 1 Assessment of the Overall Design Model 

Perceived Strengths of the UEY Design 

Finding: Key design strengths are considered to include collaboration, community 
focus, credibility and communication. 

There is limited evidence from the document and file review of the specific strengths and 
limitations of the Initiative’s design. Evidence from the Early Implementation Review, 
however, indicated that the UEY model works best when communities use a collaborative, 
inclusive approach (government and non-for-profit organization involvement), have strong 
leadership, and produce accessible information products. 

Key informants who were program officials and provincial/territorial government 
representatives and experts were asked what they considered to be the key overall 
strengths of the UEY design. Although there was no clear consensus, the following four 
categories of strengths were cited: 

• Collaboration, which includes: 

o bringing together the community’s resources and strengths with federal resources; 

o the relationship between UEY and provincial partners (e.g. assessment of proposals 
and federal/provincial/territorial committees), which helps to ensure efforts are 
complementary; and 

o HRSDC staff and the support (e.g. software) and training (e.g. for mapping) 
provided to the sites. 

• Community, which includes: 

o the use of dedicated positions (e.g. community coordinator and community 
researcher) for each UEY site rather than using existing positions within each 
community, an approach which ensures an alignment of skills with the demands of 
the UEY Initiative; 



Formative Evaluation of the Understanding the Early Years Initiative 




o community participation in data gathering (allows the community as a whole to 
develop a better understanding of the factors that help or hinder childhood 
development and promote change in the community); and 

o community capacity building through the community’s work on data collection, 
preparation of inventory of programs/services, and working with Census data to 
conduct their community mapping. 

• Credibility , which includes: 

o a sound Call for Proposals and assessment process; 

o sound data collection instruments and approaches (e.g. EDI, PIDACS, Community 
Mapping) and using independent contractors to ensure standardized data 
collection for EDI and PIDACS, while allowing for sufficient flexibility to adapt to 
community-specific needs; 

o provision of credible research data from multiple sources as a catalyst for 
community action and the development of a Community Action Plan; and 

o for the UEY Initiative as a whole, good (credible) data that has come from a wide 
array of sources and communities across Canada. 

• Communication, which includes: 

o raising awareness of the importance of early childhood development and 
encouraging a focus on children within the community; and 

o facilitating knowledge transfer. 



Perceived Challenges/Limitations and Suggested Changes for 
the UEY Design 

Finding: The main design concern identified by key informants was a lack of sufficient 
linkage between research data from the EDI and PIDACS. 

The number and breadth of the strengths of the UEY design noted by key informants 
indicates the ongoing successes and general satisfaction with the Initiative among the 
UEY sites. At the same time, key informants felt there was insufficient linkage between 
research data from the EDI and PIDACS. The UEY design initially called for the EDI 
and PIDACS to be conducted for the same cohort of children. Due to administrative 
delays in contracting, however, data collection for the PIDACS was stalled for almost a 
year. As a result, a different cohort was used for the EDI and PIDACS. Some key 
informants were skeptical that the information obtained from two different cohorts will 
provide useful and accurate data. Section 2.4.3 examines the use of two different cohorts 
in more detail. 
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Finding: The suggested improvements to the UEY design focused on improving or 
refining processes, rather then a redesign of the Initiative. Many of the suggestions 
made by the key informants (e.g. improving coordination with provincial/territorial 
early childhood development programs and projects ) have been a focus of the UEY 
Initiative from the beginning. The literature review suggested taking a longer-term 
approach and encouraging communities to also plan for the needs of older children. 

Key informants who were program officials and provincial/territorial government 
representatives and experts were asked whether they felt any changes were needed in the 
design of the UEY Initiative. For the most part, the areas they identified for improvement 
coincided with areas that have been a focus of improvement efforts from the beginning of 
the UEY initiative. Their suggestions included: 

• improve the proposal review process by devoting more effort to assessing the capacity 
of UEY applicant sites to ensure that funding goes only to communities that have the 
necessary capacity to undertake a UEY project; 

• improve coordination with provincial/territorial early childhood development projects 
and programs; and 

• explore options for improving knowledge transfer (e.g. involve the business 
community in the UEY Initiative). 

The literature review found the general characteristics of the UEY Initiative to be similar to 
many community initiatives being implemented in other countries. These characteristics 
also serve as operating principles and strategies that have been found to work well across 
those community-driven early childhood development initiatives. For example, all of the 
initiatives: 

• are locally controlled and responsive to local needs, as perceived by the people in the 
community; 

• place a priority on a community-driven process that defines program goals, intervention 
strategies, and specification of local project objectives; 

• appear to have approached their communities in a similar way (i.e. open invitation to 
participate, clear specification of participation requirements, an expectation that there 
will be key agency involvement, and the use of responsible local sponsors); and 

• support some form of evaluation. 

The literature review also found that program differences reflect value choices or 
government priorities rather than deficiencies (as indicated in Table 2.2). 
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Table 2.2 

Summary of Program Differences Across Other Programs 


Program 


Key Features 


Communities for Children 
(Australia) and Success 
by Six (United States)* 


• Focused on young children and responds to the importance of 
enriching experience for children in the preschool years. 

• Policy materials tend to refer to the importance of brain development 
and the provision of stimulating environments. 


Better Beginnings, 

Better Futures (Canada) 


• Focused on two groups: very young children (birth to age four), and 
older children (age four to eight). 


Children’s Trusts (England) 


• Concerned with all children and youth. 

• Also concerned with certain conditions (e.g. child abuse and neglect, 
poor school achievement, and anti-social behaviour in adolescents). 


Vibrant Communities 
(Canada) and Social Inclusion 
Partnerships (Scotland) 


• Whole community strategies aimed at improving the living conditions 
for all community members. 

• Aims to enhance the lives of children, adults and whole families. 


* Note that the UEY Initiative would be a part of this group of programs. 
Source: Literature review. 



Regarding suggestions for improving UEY design, the literature review noted that the 
Initiative (which is focused on younger children) varies from many similar types of 
programs in other jurisdictions in that it establishes a boundary between younger and 
older children. This suggests that the logic of the UEY Initiative might be enhanced by 
including a more explicit treatment/consideration of the issue of developmental 
continuity. For example, UEY communities could be asked to anticipate and explicitly 
plan for the needs of children after they pass beyond the early years. 

2.4.2 Assessment of the EDI 

As noted in Section 1.2.3, the EDI is a population-based measure used to rate the 
readiness to learn of four and five-year old children when they are in kindergarten. 
Data are collected from kindergarten teachers on all kindergarten children (year prior to grade 
one) in schools participating in the UEY project. Funds are required for replacement 
teachers while participating teachers attend training sessions about the completion of 
EDI questionnaire and while participating teachers complete the questionnaire for each child. 
The funds for teacher replacements are managed by the sponsoring organization, through the 
contribution agreements. 



Participation of School Boards in the EDI 

Finding: Some key informants noted that some communities experienced initial 
problems in obtaining buy-in from the school district staff administrations (e.g. some 
schools/school boards were concerned about the possible misuse of results). 

As noted in Section 1.2.5, schools/school boards are key players in implementing the 
EDI. Therefore each community’s UEY proposal had to include letters of support from 



17 Findings are reported on how children in the community are doing as a group. 
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each participating school board indicating their willingness to: a) provide lists of children 
and contact information for their families; and b) facilitate the collection of UEY data in 
their schools with their senior kindergarten (or equivalent) children. By requiring signed 
letters of commitment, the UEY Initiative sought to ensure that school boards were fully 
engaged in the UEY Initiative. 

As noted in Section 2.1.2, some community representatives and independent contractors 
indicated there were initial problems in some communities getting buy-in from the school 
district staff administrations. For example, some school boards were concerned that the 
results could be misused/misconstrued and reflect poorly on specific schools or boards. 
As well, schools in some of the provinces where the EDI had already been implemented 
(British Columbia, Manitoba and Ontario) questioned why the EDI had to be repeated. 
A few key informants also noted early challenges with respect to compensating 
schools/boards for teacher replacement costs associated with conducting the EDI. 



Perceived Strengths of the EDI 

Finding: A key strength of the EDI is that it is a standardized instrument that is 
nationally used and recognized. Other perceived strengths included processes put in 
place to help support the quality of the collected data. 

Key informants who were community representatives, independent contractors, and 
program officials were asked to comment on the strengths of the EDI. They indicated that 
a key strength is that the EDI is a standardized instrument that is nationally used and 
recognized. Certain other strengths were also identified by key informants and the 
project file review. 

• Training: providing training to teachers on how to complete the EDI was credited with 
increasing the comfort level of participating teachers, providing the basis for a more 
standardized approach for completing the EDI and mitigating (to some extent) the 
potentially subjective nature of the EDI. 

• Compensation : reimbursing schools/school boards for the time spent by teachers in 
completing the EDI was credited with increasing the level of buy-in and participation 
on the part of school boards and individual schools. 

• Research staff: having an independent contractor responsible for the EDI was 
credited with providing support to the communities and making results available in a 
timely fashion. 

• Analytical methods: the mapping of the EDI results was credited with helping to better 
illustrate the data, and increase community understanding of the data. 



18 The EDI has been widely used in some Canadian provinces (British Columbia. Manitoba and Ontario). 
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Perceived Challenges/Limitations and Suggested Changes 
for the EDI 



Finding: Most of the challenges/limitations identified for the EDI were related to the 
EDI process (i.e. certain areas of communication, reporting, and data sharing ) rather 
than related to the EDI instrument itself The suggested improvements largely reflected 
the challenges encountered with administering the EDI and were, in some cases, 
specific to the experience of the key informant’s community. 

Key informants who were community representatives, independent contractors, and program 
officials were asked to provide feedback on the limitations/challenges with the data 
collection, analysis methods and data quality of the EDI. They were also asked whether they 
felt any improvements needed to be made to the EDI data collection and analysis. 

In general, the EDI was considered to be a credible instrument and its overall 
implementation was generally viewed to have been successful. The feedback also indicated 
that some communities experienced few challenges in implementing the EDI, while others 
experienced significant hurdles. Table 2.3 shows the main challenges/ limitations related to 
the EDI process that were identified by key informants and the project file review. The 
suggestions offered by key informants for improving the EDI generally corresponded with 
these areas (as shown in Table 2.3). 



Table 2.3 

Challenges/Limitations and Suggested Changes for the EDI 


General Area 


Challenges/Limitation 


Suggested Changes 


Communication 


• Between community coordinator and 
independent contractor: there was 
insufficient interaction and a lack of 
direct contact between the independent 
contractor and community coordinators 
because communication was channeled 
through HRSDC. 


• Improve communications: to 

ensure a shared understanding 
among all key players of how the 
EDI is going to be implemented. 
Specifically the roles and responsi- 
bilities of each key player and the 
data collection process need to be 
communicated to everyone involved. 


Reporting 


• Initial dissatisfaction due to poor 
communication: a few community 
coordinators indicated that their insufficient 
interaction with the independent contractor 
resulted in initial dissatisfaction with the 
EDI reports. 

• Not sufficiently user-friendly: the early 
EDI reports were not user-friendly, and 
were difficult to understand. The reports 
have since been modified to the 
satisfaction of community coordinators. 


• Improve clarity of reporting: the 

overall clarity of the EDI reports 
should be improved. Also the reports 
should provide more explanation to 
support data tables so the reader fully 
understands the findings. 

• More information is needed on 
what communities can do with the 
EDI data/report (i.e., to help identify 
need and provide services). 


Participation 


• Participation issues for some groups 

some communities’ schools/school boards 
had ethnic or multicultural populations who 
were reluctance to participate in the EDI. 
These populations do, however, 
participate in local UEY project activities. 
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Table 2.3 (continued) 


General Area 


Challenges/Limitation 


Suggested Changes 


Data sharing 


• Data could not initially be shared: 

the letter to parents explaining the EDI 
collection was not as comprehensive as 
it might have been with respect to uses 
of data. Initial departmental review of 
the documentation and legislation 
regarding disclosure of personal 
information resulted in the projects 
being told that they would not have 
access to the personal-level EDI data 
files. A subsequent review of the 
legislations determined that in specific 
and limited circumstances the personal- 
level data can be disclosed. This 
decision resulted in a data sharing 
proposal and agreement process which 
allowed projects to receive their own 
EDI data file for the duration of their 
UEY project to do specific analysis as 
outlined in their proposal. 


• Clarify data sharing consent: the 

letter to parents (respondents) must 
be clear with respect to what is 
being collected and how the data 
will be used so that their partici- 
pation in the research is well 
informed and participating 
communities can more easily 
access the personal-level data for 
community-specific analyses. 


Length of 
assessment 


• Assessment is too long: a few 

community key informants felt that the 
EDI assessment (i.e. length of time 
needed to administer the instrument) 
was too long. 




Credibility 


• Some research-related concerns: 

some key informants were concerned 
that the EDI was conducted on only one 
cohort of children and felt that it was 
open to kindergarten teachers’ 
subjective views of individual children’s 
abilities. Program documentation 
indicated, however, that the EDI was 
selected for use based on the best 
advice of recognized academics in the 
field of early childhood development. 

• Address concerns about geographic 
and cultural biases: a few community 
representatives felt that the EDI has an 
urban and Anglo bias, which led them to 
question the validity of the results with 
respect to rural or ethnically diverse 
communities. 

• Better identification/ inclusion of 
special needs children: a few key 
informants felt that special needs children 
were being largely left out of the EDI. 




Source: Key informants and file review. 



2.4.3 Assessment of the PtDA CS 

As noted in Section 1.2.3, the parent interview component of the PIDACS provides 
information on family background and family processes, including socio-economic 
status, parenting style and use of community resources. The direct child assessments 
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examine receptive vocabulary through the Peabody Picture Vocabulary Test - Revised, 
early literacy and general cognitive skills through the “Who am I?” instrument and early 
numeracy with the Number Knowledge Test. The PIDACS was intended to be applied to 
a sample of the EDI cohort of children in each UEY community. Due to delays in 
implementing the PIDACS, however, it was applied to a different cohort. 



Participation of School Boards in the PIDACS 

Finding: Key informants indicated large variations in the level of cooperation across 
school boards. 

Key informants who were independent contractors and program officials described large 
variations in the level of cooperation across school boards. The independent contractors 
felt that the engagement and cooperation of school boards had a direct impact on the 
participation rate of parents. For example, cooperative school boards were more 
pro-active in engaging parents in the process (i.e. encouraging parental participation). 
By following up with parents, they increased the response/participation rate for the 
community. 



Perceived Strengths of the PIDACS 

Finding: A key strength of the PIDACS is that it is a validated, standardized 
instrument based on the National Survey of Children and Youth (NLSCY). Another 
key strength is that the instrument and data collection can be tailored and customized 
to accommodate differences across UEY sites. 

Key informants who were community representatives, independent contractors and 
program officials felt a key strength is that the PIDACS is a validated, standardized 
instrument based on the National Longitudinal Survey of Children and Youth (NLSCY). 
Another key strength is that the instrument can be tailored and customized to accommodate 
the differences among the 21 UEY sites (e.g. language of interviews, use of in-person 
interviews). 19 As a further strength, some key informants cited the fact that PIDACS 
collects data on both parents and children, and measures community and family factors. 

As well, certain factors were identified as facilitating the implementation of the PIDACS. 

• Flexibility in accommodating differences across communities: the process was 
adapted to meet the needs of the community (i.e. parent interviews were conducted in 
the family’s language of choice, interviews were conducted in-person when a telephone 
interview was difficult). 



19 In some communities, some interviewees preferred in-person interviews conducted at a community centre. 
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• Using local people to conduct the direct assessments with the children helped to 
increase community buy-in and helped to meet the various linguistic needs of the 
community. Local interviewers were also used for the in-person interviews in one 
community. 

• Training and workshops : community representatives found the UEY workshop on the 
PIDACS to be extremely helpful. Assessor/interviewer training was considered to be 
standardized and comprehensive. 



Perceived Challenges Encountered With PIDACS and Suggested 
Improvements 

Finding: The most frequently cited challenge was the delayed implementation of the 
PIDACS. Community representatives and independent contractors also expressed 
concern about some of the consequences of that delay. 

Most of the community representatives who were asked to comment on challenges 
mentioned the delay in conducting the PIDACS data collection. The delay was caused by 
administrative delays in implementing the UEY Initiative and in selecting the contractor 
responsible for collecting the PIDACS data. Community representatives and independent 
contractors also noted certain consequences of the delay. 

• The PIDACS and EDI were conducted on different cohorts: one consequence of the 
delay was that the EDI and PIDACS were conducted on different cohorts of children. 
UEY officials sought methodological advice on this issue and the evidence shows that 
all results from the EDI and PIDACS are valid (regardless of the use of different 
cohorts). In spite of this, comments by some key informants indicated that they remain 
sceptical and this suggests need for further dialogue with communities. 

• Collaboration with schools/school boards: confusion occurred on the part of some 
schools/school boards about their agreement to participate. Due to the delay, people 
who had agreed at the application stage had left the school/school boards by the time 
the project started. New people had to be brought up to speed on the project and 
the advantages of participation. This often involved a significant investment in time on 
the part of community coordinators. 

The following limitations were also identified: 

• Respondent burden : the parent interview is seen as too lengthy. One key informant 
reported that the parent interviews were initially about 90 minutes in length. This was 
subsequently reduced to 60 minutes, but was still seen to be too long. 

• Data sharing: UEY officials reported that a base letter was approved by HRSDC to inform 
parents about the data collection, uses and sharing of the data, and to ask for consent for 
their children to do the assessment activities and agreement to share their data with the 
sponsoring organization. It was possible for the sponsoring organizations to change 
sections of the letter that were not related to data release and sharing. In a few cases, the 
inability to revise the complete letter was an issue for some community projects. 
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Finding: Some of the suggested improvements for the PIDACS tended to be similar to 
the suggestions made for the EDI (e.g. reducing the length of the interview , clarifying 
data sharing consent, eliminating bias, providing more lead time and improved 
communication ). 

Key informants suggested a number of changes to the PIDAC that were similar to those 
suggested for the EDI. 

• Increased brevity: a few community representatives felt the interviews were too long. 

• Clarify parental approval for data sharing: use an improved/in-person administration 
of the consent form to ensure that families are not missed and parents understand how 
the information will be used. A few key informants felt that some parents may not have 
understood the consent forms that were sent, even when they were sent in the family’s 
language of choice. 

• Address concerns about geographic and cultural bias : a few key informants felt that 
the PIDACS was culturally biased and a few felt it had an urban bias. 

• More lead time, better planning and improved communication of the roles of key 
players: including a strategy that outlines roles and expectations for key players, 
timelines for implementing the PIDACS, and what contributes to success. 

Finding: The suggested improvements for the PIDACS also included certain process 
changes that related specifically to the PIDACS (e.g. adding further flexibility to 
accommodate differences across communities, and hiring local/known people as 
PIDACS administrators ). 

Key informants also suggested process changes related specifically to the PIDACS. 

• Flexibility in accommodating differences across communities: in-person interviews 
should continue to be held in the family’s language of choice so as to increase parents’ 
comfort and understanding. This is particularly important with ethnic and new 
immigrant communities. This approach has proven to be effective in communities 
where languages other than English or French are used. 

• Hiring local/known people: a few respondents felt that having PIDACS administrators 
who are known and trusted in the community increased participation rates. This approach, 
which was implemented successfully in every community, was also reported to increase 
community buy-in and sense of ownership of the UEY project. 



2.4.4 Assessment of the Community Mapping Report 

As noted in Section 1.2.4, the UEY Initiative requires each community to conduct a 
community mapping exercise to illustrate the UEY research findings for community 
members. The Community Mapping Report provides information on the linkages 
between the community’s characteristics (including socio-economic data, programs and 
services) and child outcomes. 
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Perceived Strengths of the Community Mapping Report 

Finding: Key strengths of the Community Mapping Report are the provision of useful 
data and information, and the development of local capacity to undertake research. 

At the time of the formative evaluation, most of the UEY communities had begun the 
process of community mapping, but had not completed their community mapping 
exercise. Therefore, not all community representatives felt able to comment on the 
Community Mapping Report. Those who felt able to comment, along with program 
officials, cited the following strengths related to the Community Mapping Report: 

• providing useful information : the Mapping Report provides practical and useful 
information on programs/services in the community, provides an assessment of 
program reach, and assists with the identification of overlaps and gaps in services and 
programming; and 

• capacity development: the mapping process has allowed UEY sites to develop capacity 
with respect to human resource skills and knowledge. In some cases, community 
representatives reported that UEY sites have become a “knowledge centre” for mapping 
in the region. 



Perceived Challenges/Limitations and Suggested Changes for 
the Community Mapping Report 

Finding: Challenges identified for the Community Mapping Report related to capacity 
issues within the community (i.e. a limited capacity to develop the report) and 
challenges in obtaining relevant data. 

Although there is evidence that the community mapping exercise was challenging for 
some communities, there was also a high level of satisfaction with the resulting reports 
(particularly the ability to identify available programs and services and gaps in 
programming). Certain challenges were identified within communities that had either 
completed or were in the midst of conducting the community mapping exercise. 

• Limited capacity: many UEY communities reported that they had limited capacity to 
collect and analyze data and develop the Mapping Report. Accessing the expertise 
required to do mapping was identified as a key challenge by community 
representatives and corroborated by the file review. This challenge was particularly 
apparent for small communities.' Necessary skills had to be recruited/hired from 
outside the community, which sometimes took longer than anticipated. There was no 
evidence, however, that UEY communities were unable to recruit the necessary 
capacity to undertake the mapping exercise. 



20 The community coordinator is responsible for the management of community data collection including surveys and 
research. The project sponsor decides whether the research is conducted by the community coordinator, another 
member of the UEY community, or an individual hired from within or external to the community. 
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• Data: some data points in the Mapping Report do not apply to rural areas 
(e.g. playgrounds - children in rural areas tend not to play in playgrounds). In addition, 
there were challenges in obtaining the necessary data and geographic information. 

Finding: A few key informants suggested the community mapping exercise could be 
improved by encouraging information sharing among UEY communities and by 
providing more/better guidance and assistance when communities are preparing the 
Community Mapping Report. 

A few key informants offered suggestions for improving the Community Mapping Report, 
based on their experience to date. 

• Mandatory requirements : improve the identification of neighbourhoods of a suitable 
size for UEY (i.e. so that data is not suppressed due to small sample sizes). 

• Information: improve/encourage information sharing among UEY communities. 

• Guidance/assistance: provide more guidance/assistance when communities are preparing 
the Mapping Report. Many communities lack the necessary capacity to undertake the 
mapping exercise on their own. A few community representatives noted that the mapping 
workshop provided by HRSDC was not particularly helpful/did not provide them with 
sufficient training/knowledge. 



2.5 Lessons Learned 

This section examines lessons learned by examining what has worked well for the 2005 
round of UEY sites. It also examines areas identified for change and some refinements 
made to date for the 2006 round of UEY sites. The lessons learned can be considered to 
apply to all 21 sites because the majority of data collection focused on all of these sites, 
including the document and file review and the key informant interviews with program 
officials and independent contractors. 



2.5. 1 Aspects of UEY That Have Worked Well 

Finding: Key informants identified five main aspects of the UEY Initiative that they 
felt have worked well: providing guidance and training, use of project officers, 
emphasizing community relationships/partnerships, having a community coordinator, 
and emphasizing knowledge transfer and communication activities. 

Key informants who were community representatives and program officials were asked 
to identify the key lessons they have learned to date from their experience with the 
UEY Initiative. Some reflected on the lessons they have learned with regard to the 
Initiative’s implementation, while others commented on results. Overall, five main 
aspects of the UEY Initiative were identified as having worked well (with a few key 
informants identifying each one). 
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• Providing guidance/training-, the orientation, training, and guidance provided by 
HRSDC to UEY communities was credited with providing the opportunity for sites to 
learn about the Initiative and meet UEY program management, staff and 
representatives from other UEY sites. This encouraged relationship building among 
key players involved in UEY at the national level. 

• Use of project officers : having an HRSDC project officer dedicated to each UEY site 
who is knowledgeable about the UEY Initiative was considered to be a good approach. 

• Emphasizing community relationships/partnerships', the emphasis placed on, and the 
effort put into, community relationships/partnerships for UEY sites was considered to 
be a good approach. 

• Having a community coordinator: having a dedicated position and a person who is 
know in the community as project coordinator/manager was considered beneficial. 

• Emphasizing knowledge transfer and communication activities : the emphasis and 
activities undertaken by UEY sites in these areas including booklets, Websites, 
learning sessions, workshops, newspaper articles, as well as data sharing with 
community partners. 



2.5.2 Areas Identified for Change 

Finding: Key informants identified certain process-related areas for change 
(e.g. more time is needed to plan and prepare for data collection, delays need to be 
avoided/minimized, clearer consent for data sharing, and clearer communication 
with school boards). Most of these are consistent with areas where some 
refinements ha ve been made for the 2006 round of UEY sites. 

Table 2.4 shows areas where key informants felt that changes needed to be made and 
refinements made to date. 

The review of program documents indicated that plans and program delivery mechanisms 
for the UEY Initiative are revisited regularly, with risks and opportunities for improvement 
identified and action taken as needed. Examples of improvements to UEY design and 
delivery are also shown in Table 2.4, based on the document and file review and key 
informant interviews with program officials. Overall, there is good evidence that the 
UEY Initiative is pro-active in responding to challenges or limitations in the design and 
delivery of the UEY Initiative as they arise, and in modifying aspects of program design or 
delivery as necessary. 
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Table 2.4 

Areas Identified For Changes and Refinements Made to Date 



General Area 


Suggested Changes 


Refinements Made to Date 


Planning/ 

timelines 


• More time is needed to plan and 

prepare for EDI and PIDACS data 
collection. 

• Delays need to be 
avoided/minimized: the three-year 
UEY timeline will be difficult to meet 
due to delays in project 
implementation Because of the 
delay, some buy-in was lost, 
momentum was lost in a few 
communities, and schools and 
teachers had to be re-engaged. 


• Refinements to timelines include: 
o EDI and PIDACS data collection 

are to occur during the second 
year of the project (previously 
EDI occurred in year 1 , while 
PIDACS occurred in year 2); 
o the inventory of community 
programs and services is to be 
completed in year 1 of the project 
(previously started in year 2); and 
o the development of the 
Community Action Plan starts 
in year 2 of the project. 

• Change in mandatory 
requirements: the applicant 
organization (potential project 
sponsor) must be legally 
incorporated. 


Contact with 
HRSDC 


• More contact and 

communication with HRSDC 
project analysts: a few key 
informants indicated that 
communication with their project 
analysts was insufficient. 


• Increased support to funded 
projects: providing extensive support 
to projects through templates and 
training, in addition to frequent 
contact for project monitoring. 

• Increased guidance: a financial 
reporting component has been added 
to the orientation session provided for 
new UEY sites. 

• Meeting with successful project 
sponsors: some applications 
appeared to be written by contractors 
with little input from the individuals 
who would be responsible for 
implementing the UEY project. 
Therefore, a decision was made to 
meet with successful project 
sponsors before entering into the 
next round of contribution 
agreements. This was to allow 
UEY staff to familiarize the 2006 
round of UEY project sponsors 

with HRSDC processes and 
requirements. This was also to 
better enable project sponsors/ 
sponsoring organizations to 
realistically plan their project's 
financial requirements and 
work plans. 
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Table 2.4 (continued) 



General Area 


Suggested Changes 


Refinements Made to Date 


Communication/ 

understanding/ 

participation/ 

buy-in 


• Data could not initially be 
shared: The letter to parents 
explaining the EDI collection 
was not as comprehensive as 
it might have been with 
respect to uses of the data. 
Initial departmental review of 
the documentation and 
legislation regarding 
disclosure of personal 
information resulted in the 
projects being told that they 
would not have access to the 
personal-level EDI data files. 

A subsequent review of the 
legislation determined that in 
specific and limited 
circumstances the personal- 
level data can be disclosed. 
This decision resulted in a 
data sharing agreement 
process which allowed 
projects to request their own 
EDI data file for the duration 
of their UEY project to do 
specific analysis as outlined 
in their proposal. 

• Clearer communication with 
school boards with respect to 
what is expected of them in 
terms of participation/ involve- 
ment in EDI and PIDACS. 


• Increased clarity/guidance/tools: 

tools and workshop components have 
been developed to educate and 
respond to confusion over data collections 
requirements. Specific tools (developed 
or improved) include templates for a 
comprehensive letter and consent form 
for parents involved in PIDACS data 
collection. 


Source: Document and file review and key informants. 
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3. Progress Toward Immediate Objectives 



This section examines the progress made by the UEY Initiative towards its immediate 
objectives at the time of the evaluation (i.e. approximately 18 months into the community 
projects). It also examines unanticipated impacts. As indicated in Section 2.1, the 
Initiative has three immediate objectives 

• increased support for communities; 

• increased availability and sharing of information on the role of families and 
communities in child development; and 

• increased awareness and use of parent and community knowledge regarding child 
development. 



3.1 Increased Support to Communities 

Finding: The evidence suggests that progress has been made in providing communities 
with funding support. Although it is too early to assess the ability to attract additional 
support, the UEY has enabled community coalitions to identify priorities for grants, 
incorporate UEY into funding proposals, and apply for and secure funding. 

Key informants who were UEY program officials felt that progress has been made in 
providing communities with funding support. Although the UEY objectives provide little 
indication of the anticipated sources of increased support, the quarterly activity reports 
examined by the project file review indicated that the UEY Initiative had enabled 
community coalitions to identify priorities for grants, incorporate UEY into funding 
proposals, and apply for and secure funding. 

While the ability to attract additional funding is important, most key informants who 
were community representatives indicated that it was too early to determine the impact of 
such funding. Some key informants stated that they were still gathering data, while others 
were optimistic that there would eventually be increased financial support stemming from 
the UEY Initiative. 

Finding: The key informant interviews indicated that progress has been made in 
providing communities with technical assistance, with community mapping being 
cited as a main catalyst for this progress. 

Key informants who were community representatives and program officials felt that 
progress has been made in providing communities with technical assistance. 

Most key informants who were community representatives reported that there was 
increased technical assistance for their communities as a result of their involvement with 
the UEY Initiative. They cited community mapping as the main catalyst for this progress. 
Progress in some communities included consultation on data collection and use, and 
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knowledge dissemination. In other communities mapping was cited as raising awareness 
of the need for technical support. The evidence also suggests that the community 
mapping exercise, while challenging for a few communities, has resulted in the 
development of capacity within communities. 



3.2 Increased Availability and Sharing of Information 
3.2. 1 Partnerships 

Finding: Evidence from the file review and key informant interviews indicated that 
UEY communities have been successful in forming or strengthening partnerships as a 
result of the Initiative. 

Key informants who were community representatives and program officials generally 
believed that partnerships have increased and strengthened as a result of the UEY Initiative. 
The review of project files also indicated that UEY communities have been successful in 
forming or strengthening partnerships. For example, information drawn from the project files 
indicated that all UEY sites have formed partnerships, ranging from 19 partnerships in one 
community to one partnership (with an average of ten). As noted in Section 1.3.2, however, 
the available community partnership data should be treated with caution because problems in 
determining what constituted a partnership makes it difficult to interpret these data. 

Finding: The most frequent partnership reported was with schools/school boards/school 
districts, followed by municipal and provincial governments, community service 
providers, and health and social service networks. 

Table 3.1 provides a summary of the types of organizations involved in partnerships with 
UEY communities. Table 3.1 also shows the number of UEY communities that reported a 
partnership with each type of organization. These data were collected from the UEY project 
files and should be treated with caution (as noted above). The most frequent partnership 
reported was with schools/school boards/school districts (reported by 17 UEY sites), 
followed by municipal and provincial governments, community service organizations, and 
health and social service networks. Other types of organizations reported to be involved 
in partnerships included other not-for-profit/non-governmental organizations and other 
UEY sites and colleges. 21 



It should be noted that further information would be needed to explore the nature of these partnerships. 



Formative Evaluation of the Understanding the Early Years Initiative 




Table 3.1 

Organizations Partnering with UEY Sites 


Type of Organization 


Number of UEY Sites 
Reporting Partnership 


Schools/school boards/districts 


17 


Municipal government 


16 


Provincial government 


15 


Community service providers 


14 


Health or social service networks 


13 


Other not-for-profit/non-governmental 


11 


Other UEY sites 


10 


Colleges 


9 


Researchers/research institutes 


7 


Aboriginal organizations 


6 


Universities 


4 


Religious organizations (e.g. local church) 


3 


Parents/parents groups 


3 


Private sector 


3 


Federal government (other than HRSDC) 


2 


Other (specify) 


5 


Source: UEY Project files, 2007. 



While there is evidence that partnerships have increased, at the time of the evaluation 
there was no evidence of an impact on the level of support to families with children. 



3.2.2 Networks and Mechanisms to Share Information 

Finding: Evidence from the file review suggests that networks and mechanisms have 
been developed to share information between HRSDC and UEY communities. 

Evidence from the file review indicates that there has been increased sharing of 
information between HRSDC and UEY communities. According to the project files, 
15 of the 21 UEY sites reported participating in networks or mechanisms to share 
information with HRSDC. 

Finding: Evidence from the document and file review and key informants suggests that 
the UEY Initiative has increased the number of opportunities and tools for knowledge 
sharing and networking within communities on the role of families and communities 
in child development, with the types and number of activities varying across UEY sites. 

The UEY Initiative provided assistance and guidance on sharing information within the 
community. For example, the Initiative has developed a guide for communities entitled 
Sharing the Knowledge: Maximizing the Impact of Your Project. 

Most key informants who were community representatives and program officials agreed 
that there has been progress towards more opportunities to share knowledge and network 
within communities. One community sponsor commented that “I think this has been the 
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biggest bang for the UEY buck so far.” Key informants cited a variety of mechanisms 
through which knowledge sharing and networking were taking place (e.g. task forces, 
Websites, presentations, workshops, various collaborations and the local media). 

The file review indicated that all UEY sites reported either developing or participating in 
networks/mechanisms to share information within the community. The file review also 
indicated that information sharing activities have varied across sites (see Table 3.2), but 
often included attending conferences (reported by 19 UEY sites), presentations to school 
boards/school districts (reported by 15 UEY sites), and participation in relevant local 
committees (reported by 15 UEY sites). 



Table 3.2 


Types of Information Sharing Activities 


Information Sharing Activity 


Number of UEY Sites Reporting 
Participation in Information Sharing Activity 


Conference attended 


19 


Presentations to school boards/districts 


15 


Participation in relevant local committee 


15 


Presentations on UEY 


11 


Sharing of EDI data with school boards/districts 


11 


Participation in community planning sessions 


11 


Community consultations 


10 


Participation in community events 


10 


Briefings of community leaders on UEY 


9 


Other (specify) 


8 


Presentation at a conference 


7 


Community information session/open house 


5 


Site visits/presentations to local communities 


5 


Hosting public event related to UEY 


3 


Meet with/present to children and family 


3 


Interviews with media 


2 


Distribution of UEY research reports 


1 


Presentation/guest lecture at university 


1 


Source: UEY project files 



Most key informants who were community representatives were able to cite a variety of 
networks, mechanisms and tools that had been created in the community to share 
information on child development since the beginning of the UEY project. Examples 
included Websites, community meetings, reports and research communiques showing 
results, and surveys. Although the PIDACS and EDI were frequently identified as being 
key to the increased knowledge and data, key informants also identified other documents 
and information that had been developed and shared with the community at large. 

The file review also indicated that UEY communities have produced a variety of 
knowledge dissemination tools (see Table 3.3). The most frequently cited tools were 
resource guides/asset lists (reported by 15 UEY sites). Websites or Web pages to present 
UEY information (reported by 14 UEY sites), and PowerPoint presentations (reported by 
12 UEY sites). The document review and key informant interviews indicated that much 



Formative Evaluation of the Understanding the Early Years Initiative 




of the work done to date can be considered groundwork (i.e., getting the project and those 
involved known in the community at large). In the first quarter of 2007, a number of 
activities related to knowledge products began to take place. 



Table 3.3 

Types of Knowledge Dissemination Tools 


Knowledge Dissemination Tool 


Number of UEY Sites 
Reporting Production of Tool 


Resource guide/asset list 


15 


Website/webpage to present UEY information 


14 


PowerPoint presentation 


12 


UEY handout/brochure 


10 


UEY newsletter 


10 


Press release 


10 


EDI handout/brochure 


9 


UEY article 


8 


Social marketing plan/communications strategy 


8 


Posters 


8 


Letterhead 


5 


EDI reports 


4 


Research communique 


3 


Community Mapping Report 


3 


Resource library 


1 


Other (specify) 


14 


Source: UEY project files 



3.3 Increased Awareness of Child Development 

Finding: The evidence suggests that the UEY initiative has increased parental and 
community awareness and knowledge regarding child development to some extent, 
although it was too early for this progress to be fully evident. 

Most key informants who were program officials believed there has been progress in 
terms of increasing parental and community awareness and knowledge regarding child 
development. They cited the creation of brochures and presentations at workshops as 
examples of progress. Most key informants who were community representatives 
indicated there has been at least some progress in parental and community awareness, 
although some felt it was too early to assess this area. 22 Many cited increased dialogue 
with parents as an example of progress. 



' 2 Many communities had only recently begun to receive data and develop the necessary tools and mechanisms for 
sharing information with their community. As well, it is reasonable to anticipate that it will take time before the 
information from UEY is synthesized or communicated to parents and caregivers. It can also be argued that the 
achievement of this immediate objective is dependent on the achievement of the Initiative’s other immediate 
objectives, because there cannot be increased awareness and knowledge without the availability and sharing of 
information. 
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Finding: At the time of the formative evaluation, the evidence was inconclusive on 
whether there was increased use of knowledge regarding child development. 

Most key informants who were program officials did not know if progress was being 
made towards increased parental and community use of knowledge regarding child 
development. Some said it was too early to tell. These key informants did see progress in 
facilitating the exchange of information on child development, however. As examples, 
they cited information sharing events and brochures. 

Key informants who were community representatives had mixed opinions on whether 
there has been increased parental and community use of knowledge. They felt that this 
was difficult to determine and that it was still early in the local UEY projects’ lifespan. 
A few noted that parents are seeking out more information, which suggests that parents 
are using this information. 

The evaluation team noted that program documentation and files for this indicator were 
limited. The number of requests for UEY information on early childhood development 
does not appear in project files. As well, as indicated in Section 3.2.2, the types and 
number of tools and activities for knowledge transfer/information sharing varied across 
UEY sites. 



3.4 Unanticipated Impacts 

Finding: Key informants identified a number of unanticipated impacts. 

Key informants who were program officials, community representatives, independent 

contractors, and provincial/territorial government representatives and experts identified 

the following positive unanticipated impacts: 

• community coordinators and coalitions have assisted with other social development 
activities within the community beyond the scope of early childhood development; 

• issues of child care, in general, have been brought to the forefront; 

• some community coordinators are now seen as experts in the community in other areas 
relevant to UEY ; 

• there is broader awareness of and interest in the early childhood issues (e.g. more 
requests for information) than expected; 

• there is increased ability within the community to engage local leaders; 

• there has been knowledge sharing with other communities involved in the UEY Initiative 
who have the same motivation and vision; and 

• non-funded applicants often felt that the UEY proposal process helped them to bring the 
community together and form networks that continue to do work on early childhood 
issues even in the absence of funding. 
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The key informants also identified the following negative unanticipated impacts: 

• because of delays, EDI and PIDACS data collections involved different cohorts of 
children leading some to question the credibility of results; 

• demands on community coordinators’ time (e.g. to make presentations to other audiences) 
are a significant burden; 

• the late recognition of the implications of privacy legislation meant that communities 
were initially offered access to aggregate data only (which limits local analysis and 
understanding). A subsequent review of the legislation determined that in specific and 
limited circumstances the personal-level data can be disclosed. This decision resulted 
in a data sharing proposal and agreement process which allowed projects to request 
their own EDI data file for the duration of their UEY project to do specific analysis as 
outlined in their proposal; and 

• requests for administrative assistance in conducting the PIDACS overwhelmed some 
school boards. 

The document and file review did not provide any evidence of unanticipated impacts 

(either positive or negative). 
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4. Accountability Issues 



This section examines: 

• the adequacy of program safeguards to avoid duplication; 

• the appropriateness of reporting strategies; and 

• the adequacy of reports for program monitoring. 

The availability of suitable data for the summative evaluation is examined in Section 5. 



4.1 Adequacy of Safeguards to Avoid Duplication 

Finding: Despite the array of provincial programs aimed at children , the program scan 
conducted for the formative evaluation found no evidence of overlap or duplication 
with the UEY Initiative. 

The literature review included a scan of programs in Canada. The scan did not reveal any 
universally available programs that would be jeopardized by the UEY Initiative. 
For example, programs identified as having a similar intent or approach (Success by Six, 
Better Beginnings, Better Futures, and Vibrant Communities) were available only on a very 
limited basis or were dependent on voluntary funding sources or ‘soft’ money. Most other 
early childhood initiatives (such as Best Start in Ontario) take quite different approaches and 
focus on different objectives. 

Finding: A number of safeguards have been put in place to avoid or lessen the 
potential for overlap and duplication. 

A number of steps have been taken to avoid or reduce the potential for overlap and 
duplication. 

• The Federal/Provincial/Territorial (FPT) Committee on Early Childhood Development 
Knowledge Information and Effective Practices meets twice annually and helps to 
ensure that FPT activities are coordinated and that duplication is avoided. 

• UEY sites are required to inform the program analyst if they receive funding from any 
other federal, provincial or municipal sources that duplicate UEY funding (i.e. cover 
the same activities). If the other funding is received, the site’s UEY funding is adjusted 
to prevent duplication. 

• The external review process includes asking external assessors if they are aware of any 
UEY-like or UEY-related activities planned for the community such as the collection of 
data similar to UEY, data mapping work, and coalition building through other projects. 



23 Key informants who were program officials reported that the Initiative does not duplicate any other federal program. 
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As well, UEY officials were confident that their own activities through networks and 
conferences help to keep them informed about other programs or initiatives being 
discussed or developed. All key informants who were program officials expressed 
confidence that the safeguards are adequate and do not need improving. 

Finding: Evidence indicates that miscommunication was at the root of situations where 
some overlap and duplication may have occurred. This suggests a need for better 
communication between community representatives and program officials prior to 
project implementation. 

Unlike their federal counterparts, most key informants who were provincial/territorial 
government representatives and experts were unaware of the safeguards in place to prevent 
overlap and duplication. Some cited examples of conflict. For example, one key informant 
explained how a provincial evaluation study competed directly with a UEY project for the 
time and resources of teachers and parents." As noted above, however, community 
proposals for UEY funding are expected to include information about other programs or 
projects that may compete with or duplicate the UEY Initiative. However, most key 
informants noted that one key way to avoid overlap and duplication is to maintain clear 
communication channels (especially in times of staff turnover). 



4.2 Appropriateness of Reporting Strategies 

Finding: The general view among key informants is that the results-based reporting 
strategies for the UEY Initiative are appropriate and compatible with federal 
government accountability requirements. 

Based on a suggestion in the Early Implementation Review, a Results-based Management 
Accountability Framework (RMAF) has been developed for the UEY Initiative. All key 
informants who were program officials expressed confidence that the required reporting 
strategies are appropriate and effective, and are compatible with federal government 
accountability requirements for contribution agreements. Most key informants who were 
community representatives believed that the results-based reporting strategies are 
appropriate and effective. 

Finding: Most key informants who were community representatives expressed 
confidence in their community’s capacity to undertake the necessary reporting, 
although most also expressed the desire for more clarity and feedback from HRSDC as 
they moved forward with reporting on the results of data collection. 

All UEY communities are required to submit quarterly activity reports in order to receive 
funding and to be reimbursed for project activities. In addition, the Initiative has 
developed a training session for reporting on UEY project results. This training 
session/guide is intended to encourage consistency in project reporting. 



24 One key informant cited a case where the boundary of an UEY site overlapped with a provincial early childhood 
development project site. This resulted in the provincial program being unable to measure results attributable to the 
provincial project in that community. Problems encountered in conducting an evaluation are not usually considered 
to be examples of overlap and duplication, however. 
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Key informants who were community representatives were asked to comment on what 
reporting they have done on their project’s results for UEY. Some indicated that they 
would not be able to proceed with reporting on the results of data collection until the 
Community Research Report was made available to them in January 2008. Most, 
however, listed a number of reporting activities that went beyond the required quarterly 
activity reports. Most also expressed confidence in their community’s capacity to 
undertake the necessary reporting. 

Key informants who were community representatives also viewed the reporting strategies 
(quarterly reports) as effective in terms of helping to track their own progress, and in 
providing HRSDC with oversight. At the same time, most expressed the desire for more 
clarity and feedback from HRSDC as they moved forward with reporting on the results of 
data collection. This was especially true for those facing timeline challenges. That group 
also suggested that it would be helpful to have HRSDC provide clearer and more 
reasonable reporting expectations. 



4.3 Adequacy of Reports for Performance Monitoring 

Finding: Although all key informants who were program officials felt that the 
performance monitoring reports of UEY sites were being sent in a timely manner, the 
project file review found that two-thirds of project files were missing at least one 
quarterly report. 

All key informants who were program officials felt that the reports for ongoing 
performance monitoring of UEY sites were being received and usually in a timely 
manner. The project file review found, however, that two-thirds of UEY project files 
were missing at least one quarterly report. 

The program officials were satisfied that the information in the reports is useful and 
sufficient for performance monitoring. One explained that HRSDC staff follow up and 
get the necessary information (i.e. a narrative description of progress and financial 
information) if the initial information does not appear to be useful. 
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5. Guidance for a Summative Evaluation 



This section provides some preliminary suggestions/guidance for the design of a 
summative evaluation of the UEY Initiative. These suggestions could serve as input to 
the development of the evaluation framework for the summative evaluation. This section 
also provides some data-related suggestions to better support the summative evaluation. 

Finding: A number of features of the UEY Initiative (e.g. a multi-site funding 
framework with community-level interventions) were identified as important 
considerations for conducting a summative evaluation. 

The evaluation team conducting the formation evaluation identified a number of key 
features of the UEY Initiative that can be expected to have a strong bearing on what 
approaches can be used to conduct a summative evaluation. 

• The UEY Initiative is not typical of the types of programs referred to in the program 
evaluation literature. Instead, it is a funding framework designed to provide communities 
with local data and the capacity to develop information as a way to identify gaps in 
services and programs and to foster partnerships pertaining to child development. 

• The UEY Initiative recognizes that families and children in different communities face 
different challenges. The project activities that result from the Initiative will be 
uniquely designed to meet the specific needs of each community, but may still show a 
high degree of common content across communities. 

• The UEY Initiative is designed to permit a high degree of local control over program 
features and operation. This makes it difficult, on an a priori basis, to specify what the 
particular outcomes will be and how they ought to be assessed. 

• The Initiative is grounded in current psychosocial theory in seeing child and family 
outcomes as the product of complex social system processes where many cultural and 
community effects combine to determine outcomes for individuals (children). 

These features of the UEY Initiative indicate that the approach developed for the 
summative evaluation will need to take into account the uniqueness of each community 
and the impacts of the UEY Initiative on each community as well as the overall impacts 
of the Initiative. This strongly implies that evaluation strategies will need to be 
implemented at the level of each community in anticipation of the summative evaluation 
to ensure that outcome data are available at the community level as well as at the level 
the overall Initiative. 
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Finding: After taking into account the features of the UEY Initiative and the 
uniqueness of each community, a theory of change approach 25 appears to be the most 
appropriate approach for the summative evaluation. 

The literature review identified evaluation models appropriate for multi-site national 
human service programs that share many of UEY’ s features. 26 Seven different approaches 
to conducting evaluations were considered to be particularly relevant to the needs of 
multi-site evaluations: cluster evaluation, developmental evaluation, extended-term, 
mixed method evaluation, five-tiered approach, initiative evaluation, systems approaches 
in evaluation, and theory of change. Based on her review, Dr. Lilley concluded that a 
theory of change approach was most appropriate for national, multi-site human service 
program initiatives and particularly for those involving community-level interventions 
with a strong degree of local control over the interventions implemented at the 
community level. 

Finding: The evaluation team also suggested ways to better support the summative 
evaluation (e.g. do more to ensure that baseline data are collected consistently ). 

The evaluation team examined the availability of data for conducting a summative 
evaluation based on findings from the preliminary consultations and the review of data 
collection capacity conducted to help plan the formative evaluation. They found that not 
all of the necessary baseline data is being collected consistently. For example, there is a 
need to more clearly define information/data being requested from the UEY sites (e.g. 
clarify what is meant by partnerships). As well, baseline measures identified in the 
Performance Measurement Framework in the RMAF for the UEY Initiative should 
ideally be designed to be on-going, rather than providing a single snapshot. 

The evidence also indicated a need to refine the logic model and more clearly articulate a 
set of measurable outcomes and indicators. Expected outcomes should be sufficiently 
focused so that changes may be attributed to the Initiative. As well, there is evidence that 
some outcomes are too broad in scope to be influenced by UEY (as noted in Section 2.1). 



25 See footnote number 2 on page v for the definition of Theory of Change. 

26 Lilley, S. (2006). Literature Search on Models for Evaluation of Multi-Site National Programs such as CAPC and 
CPNP. Prepared for the National Evaluation Team for Children (NETC), Population Health Agency of Canada. 
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6. Overall Conclusions 



This section highlights the overall conclusions provided by the formative evaluation. 
It also highlights the main areas identified for consideration/improvement. 



6.1 UEY Design and Implementation 

The formative evaluation found the design of the UEY Initiative to be generally sound. 
The evidence indicated that the UEY model is inclusive of a broad range of communities 
and is sufficiently flexible to accommodate their participation. Key strengths of the UEY 
design are considered to include collaboration, community, credibility and communication. 

The objectives and intended outcomes of the UEY Initiative are consistently communicated, 
and participants appear to have a clear understanding of their project’s immediate objectives 
and expected results for the UEY Initiative. At the same time the focus and measurability of 
some of the Initiative’s intended outcomes could be improved, particularly in the case of the 
longer-term and ultimate outcomes. 

The main design concern identified by key informants was a lack of sufficient linkage 
between research data from the EDI and PIDACS. The UEY design initially called for 
EDI and PIDACS to be conducted for the same cohort of children. Due to delays in 
contracting, however, data collection for the PIDACS was stalled for almost a year. As a 
result, a different cohort was used for the EDI and PIDACS. Although HRSDC has 
assured the communities that this will not affect the usefulness of the data, some key 
informants continue to be skeptical that the information obtained from two different 
cohorts will provide useful and accurate data. 

The suggested improvements to the UEY design focused on improving or refining 
processes (e.g. improving coordination with provincial/territorial early childhood 
development projects and programs) rather than a redesign of the Initiative. The literature 
review suggested that the Initiative could be improved by taking a longer-term approach 
and encouraging communities to plan also for the needs of older children. 

All 21 UEY communities have made progress toward implementing their project and 
are making progress towards their immediate objectives and expected results for the 
UEY Initiative. Although most sites experienced some early difficulties, those 
difficulties were largely a result of the challenges/characteristics within individual 
communities (e.g. staffing challenges and challenges in obtaining buy-in from certain 
groups and school boards), rather than a result of specific flaws in the UEY design. 

The roles and responsibilities of key players in the UEY Initiative were well understood 
at the time of the evaluation, and have been implemented as intended, although there was 
some need for clarification early on. Some evidence of variability in the implementation of 
roles and responsibilities across UEY communities was noted, but those variations were 
within the parameters set out by the UEY Initiative. 
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The EDI, PIDACS and the Community Mapping Report are generally positively 
regarded by all key players in the UEY Initiative. At the same time, however, some 
process-related challenges/limitations were identified for each of these UEY research 
tools, along with some suggestions for change. 

• For example, as noted above, there was concern about the use of different cohorts of 
children for the EDI and PIDACS. 

• As well, the late recognition of the implication of privacy legislation and lack of clarity 
in parental information letters meant that access to EDI data could not occur initially as 
originally anticipated. At the time of the interviews, community representatives 
continued to express disappointment over the lack of access to individual level data. 
However, a subsequent review of the legislation determined that in specific and limited 
circumstances the personal-level information can be disclosed. The decision resulted 
in a data sharing proposal and agreement process which allowed projects to request 
their own EDI file for the duration of their UEY project to do specific analysis as 
outlined in their proposal. 

• Some communities’ schools or school boards had ethnic or multicultural populations 
who were reluctant to participate in the data collection process (EDI and PIDACS). 
These populations do, however, participate in local UEY project activities. 

• Some communities have experienced challenges in finding the necessary capacity to 
develop the Community Mapping Report, although there is good indication that all 
communities either have or will be able to recruit the necessary capacity. 

The document review indicated that the UEY Initiative has been pro-active in responding 
to challenges stemming from the design and delivery of the UEY Initiative. This has 
included working to resolve process-related issues in the areas of planning/delays, 
clarity/guidance/tools and communication. 



6.2 Progress Toward Immediate Objectives 

The early evidence suggests that progress is being made toward the three immediate 
objectives of the UEY Initiative. 

Increased Support for Communities 

Evidence from key informants and the file review suggests that progress has been made 
in providing communities with funding support. Although it is too early to assess the 
ability to attract additional support, the file review indicated that the UEY Initiative has 
enabled community coalitions to identify priorities for grants, incorporate UEY into 
funding proposals, and apply for and secure funding. The evidence also indicated that 
progress has been made in providing communities with technical assistance, with 
community mapping being cited as a main catalyst for this progress. 
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Increased Availability and Sharing of Information 

Both the project file review and key informants indicated that UEY communities have 
been successful at forming or strengthening partnerships as a result of the Initiative. 
As well, evidence from the document and file review and key informants suggests that 
the Initiative has increased the number of opportunities and tools for knowledge 
sharing and networking within communities on the role of families and communities 
in childhood development. There was also evidence of the sharing of information among 
UEY communities. 



Increased Awareness Regarding Child Development 

Most key informants who were program officials felt there has been progress in terms of 
increasing parental and community awareness and knowledge regarding child 
development , although it was too early for this progress to be fully evident. Cited 
examples of progress included the creation of brochures and presentations at workshops. 



6.3 Accountability Issues 

Despite the array of provincial programs aimed at children, the program scan 
conducted for the formative evaluation found no evidence of overlap or duplication 
with the UEY Initiative. In addition, a number of safeguards have been put in place to 
avoid or reduce the potential for overlap and duplication (e.g the Federal/Provincial/ 
Territorial Committee on Early Childhood Development Knowledge Information and 
Effective Practices). 

The general view among key informants is that the results-based reporting strategies 
for the UEY Initiative are appropriate and compatible with federal government 
accountability requirements. Most key informants who were community representatives 
expressed confidence in their community’s capacity to undertake the necessary reporting, 
although most also expressed the desire for more clarity and feedback from HRSDC as 
they move forward with reporting on the results of data collection. The file review found, 
however, that two-thirds of project files were missing at least one quarterly report. 



6.4 Guidance for the Summative Evaluation 

Certain features of UEY were identified as important considerations for the design of the 
summative evaluation. For example, the Initiative is not typical of the types of programs 
referred to in the program evaluation literature. Instead it is a funding framework 
designed to provide communities with local data and capacity to develop information as a 
way to identify gaps in services and program and to foster partnerships pertaining to child 
development. As well, the Initiative is designed to permit a high degree of local control 
over project features and operation. 
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These aspects of UEY Initiative indicate that the approach developed for the summative 
evaluation will need to take into account the uniqueness of each community and the 
impacts of the Initiative on each community as well as the overall impacts of the UEY 
Initiative. This strongly implies that evaluation strategies will need to be implemented at 
the level of each community in anticipation of the summative evaluation to ensure that 
outcomes data are available both at the community level as well as at the level of the 
overall Initiative. 



6.5 Main Areas for Consideration/Improvement 

The formative evaluation has identified the following main areas for consideration/ 

improvement: 

• Some of UEY's long-term objectives may be too broad relative to the scope and size of 
the UEY Initiative. These objectives should be clarified or modified to better link them 
to the resources and scope of the UEY Initiative. 

• The UEY Initiative is working to resolve process-related issues (e.g. related to planning, 
delays) and this work should continue. 

• Consider streamlining parent interviews. 

• The linkage of research data from EDI and PIDACS may better enable research on 
possible relationships between early childhood outcomes and a number of family and 
community factors. As well, there is a need for program officials to better communicate 
with community coordinators and others in the community about the cohort used for the 
PIDACS and the usefulness of these data. The continued concern expressed by 
community representatives in this area indicates the need to validate the PIDACS results 
once they are all available. 

• It would be helpful to further assist communities in completing the Community Mapping 
report. This could be accomplished in different ways (e.g. additional in-person training 
sessions, an on-line training module or a helpline). 

• To better support the summative evaluation, the Initiative needs to consistently collect 
data at the community level on how knowledge gained within the UEY communities is 
being applied regarding child development. Guidance will need to be provided to the 
communities on how best to collect these data. 

• To better support the summative evaluation, the UEY Initiative and the UEY sites 
should ensure that outcomes are linked to measurable indicators and that the necessary 
data can, and are, consistently collected. Consideration should be given to improving 
the activity report template to better facilitate the quantification and analysis of 
UEY project progress/performance. As well, there is a need for improved quantitative 
baseline data for indicators of intermediate and longer-term outcomes. 
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Appendix A - UEY Communities 



Table A:1 

The 21 UEY Community Projects Funded in the Fall of 2005 


Province 


UEY Community 


BC 


UEY Greater Victoria 
UEY Mission 

UEY Okanagan Similkameen 
UEY Sunshine Coast 
UEY Campbell River 

UEY North Shore (North and West Vancouver) 


SK 


UEY Northeast Saskatchewan 


MB 


UEY Division scolaire franco-manitobaine 


ON 


UEY Niagara Region 
UEY Ottawa 

UEY Northern Region of Ontario 

UEY Kawartha Lakes & Haliburton County 

UEY Lower Hamilton 

UEY Northumberland County 

UEY Milton 


QC 


UEY Pointe-de-l’lle 

UEY Montreal Chassidic and Orthodox Community 


NB 


UEY Greater Saint John 


NS 


UEY Cumberland County 
UEY Western Nova Scotia 
UEY Halifax West and Area 


Source: Document and file review. 



A second group of 15 communities is being added to the Initiative from the 2006 Call for 
Proposals. 
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